MySQL Batch Create's Awesomeness

Awhile back I had to insert tons of data into a MySQL database. The app had already been built and using a different database was not an option. The dataset I was creating was massive. I needed to create 100k or more records.

I attempted to do this with a nice clean ActiveRecord statement:

100000.times do |record|
  Record.create!({
    attribute_1: "attribute 1",
    attribute_2: "attribute 2",
    ...
    attribute_20: "attribute_20"
  })
end

The results were not pretty. Although I didn't hit any timeouts, the query was taking HOURS - and likely would have timed out on a server that had less memory than my VM. This was not going to work because this was just one part of a module I was writing that needed to run.

I spent a lot of time researching creating lots of records at once and decided there was only one conclusion — the solution was MySQL batch insert. Note: this method will not work if you have complicated callbacks that need to run after each record is created. Here's how I handled the situation:

# setup the array to hold the values for the batch insert
record_values = []

# for this case we will pretend we have an array that needs to be sorted into readable data
record_data.each do |record|
  record_values << "('#{record.something}','#{record.something_else}','#{Time.now}'...'#{record.attribute_20})"
end

# set the number of records you want MySQL to handle at once. Too many records and MySQL will run out of memory
batch_size = 20000

# create a while loop to batch insert the records
while !record_values.empty?
  # use shift to move down the array based on batch size
  record_values_shitfted = record_values.shift(batch_size)
  
  # create the required sql string
  record_values_sql = "INSERT INTO records(attribute_1, attribute_2, attribute_3...attribute_20) VALUES#{record_values_shifted.join(", ")}"
  
  # execute sql
  ActiveRecord::Base.connection.execute(record_values_sql)
end

This was infinitely faster than ActiveRecord could be and unfortunately it doesn't have a way to do a batch insert. Sometimes straight up MySQL is faster and better than ActiveRecord.

Categories: activerecord mysql

Forwarding Domains with Nginx

Recently I decided to switch the url of this blog from eileenbuildswithrails.com to eileencodes.com. If you're curious about why I changed the name read this post.

With this change I wanted to keep eileenbuildswithrails.com, but forward it to eileencodes.com. Since nginx handles domain names based on alphabetical order the default domain was not forwarding correctly. The key is to forward the domains in the config files themselves.

Setting up the redirect in nginx is really easy once you know how to do it. Simply set up two server definitions; one to forward the old domain to the new domain, and one to handle rewrite of the new domain to www. And that's it. See below:

server {
  listen 80;
  server_name olddomain.com www.olddomain.com;
  rewrite ^(.*) http://www.newdomain.com$request_uri permanent;
}
server {
  listen 80;
  server_name newdomain.com;
  rewrite ^ http://www.newdomain.com$request_uri permanent;
}
Categories: nginx

Name Changes, Domain Changes, Oh My!

Those of you who know me and follow me on twitter know I got married recently. I decided to take my husband's last name and that gave me a little bit of an identity crisis. I started looking at all my usernames and realized my internet identity was a tad spastic. Some places I have my maiden name, others I use buildswithrails. I figured it was time to start unifiying everything. Although buildswithrails has been good to me I've decided to get everything under one easy name, so I went with "eileencodes".

It's short, it's sweet, and it's easy to remember. Most of all it doesn't rely on a coding language or my last name to identify myself. So now I am eileencodes on twitter, at my blog, and on github. It will be easier for other people to know who I am and I won't feel like I need to explain my life story when asked for my handle.

Categories: updates

An ActiveRecord Assumption: Using delete_all on a Related Model, StackLevel Too Deep

Recently I've been working with larger datasets than I previously have in the past, which has led me to learn much more about how ActiveRecord handles mysql queries. When using a gem or a library that does a lot of behind the scenes magic, there is a tendency to rely on that magic; until it bites us in the ass. I got myself into such a situation with delete_all on a related model.

I changed a bit of our code in the PhishMe app to use delete_all instead of destroy_all or a method we previously used to destroy in batches. I wanted this process to be faster and since we didn't need the callbacks delete_all would work perfectly.

I expected that the call for Model.related_mode.delete_all to produce the following mysql query:

DELETE FROM `related_model` WHERE `related_model`.`id` = '1'

In fact, the produced mysql that was created by the ActiveRecord query was:

DELETE `model` WHERE `model`.`id` = '1' AND ((`related_model`.`id` = '2'<br />OR `related_model`.`id` = '3' OR `related_model`.`id` = '4' OR<br />`related_model`.`id` = '5' OR...))
 13 

The above query normally wouldn't cause any issue and you might not even noticed the slowed down query, but if you have thousands of records it blows up, and not gracefully.

When you hit the threshold of the number of records that can be chained, ActiveRecord spits out a "StackLevel too deep". (facepalm) After I thought about it, the issue was obvious. The stack really is too deep. I did actually ask mysql to chain all the methods. I didn't expect this behavior because I had never considered the ramifications of the Model.related_model.delete_all call I was making.

I'm becoming very interested in how our ActiveRecord calls translate into mysql queries. It's easy with "magic" gems to ignore the underlying behavior.

Lessons Learned from Planning a Wedding in the "Digital Age"

For the last six months or so, my life has been consumed by non-ruby stuff. In June I got married and during the 10 months leading up to the event, the time I would have otherwise devoted to side projects and ruby learning, was entirely consumed by what my husband and I worked hard to make a very smooth and tech-savvy process. We learned many lessons along the way, and I have decided to share some words of wisdom with other technology hungry folks who may be planning a wedding as well.

Lesson 1: RSVP Cards are Necessary and a Website Form Doesn't Cut It

I thought I would save a few trees, spend a little less money on stamps and RSVP cards so I decided to make an online form for the people who had a computer and knew how to use it. I figured it would make it easier for my family and friends to RSVP and their card wouldn't get lost in the mail.

This brought up a few issues I didn't anticipate. First, my friends and family couldn't remember that they didn't RSVP yet because there was no card to send in — they had nothing to sit on their desk indicating that they hadn't RSVP'd yet. This memory lapse only got worse the more they procrastinated. Secondly, the form led to confusion about who was actually invited. Because the form tried to lend itself to all types of guests, they didn't know if they could have plus ones, etc.

If you do use an RSVP form in lieu of RSVP cards, make sure to send them back a "thanks for RSVPing" so they're more likely to remember they already RSVP'd or add a reminder email for those who haven't filled out the form yet.

Lesson 2: Don't Give Your Email Address or Phone Number to Any Company Ever, Unless They Are One of Your Vendors

This one is just a good ordinary life lesson. Of course you can give your information to your vendors, but only after you've decided who your vendors are so they can't contact you when you ultimately decide to go in a "different direction".

I found that large corporate vendors tend to hire people who work only on commission and are desperate to make some kind of sale. As a result, I got countless phone calls and emails from people I never asked to get information from.

Many wedding sites say in their "Terms" (TheKnot for example) that even if you opt-out of them selling your information, they might do it anyway and the only way to stop them is to contact them directly. It's really disheartening that companies conduct themselves in this way.

Lesson 3: Definitely Use Trello or Another Project Management Tool for the Planning Process

Using Trello for wedding planning was one of the best decisions we ever made. Abe and I used it to assign each other tasks (ex. "Please call florist."), make sure we knew when deposits were due and who we had already paid. We even used it to organize our seating chart. Each family member/friend was assigned a color based on the side of the family they were associated with. This made it really easy to know who could sit with who.

Trello has a really great drag and drop interface that makes it easy to organize projects or weddings. Abe goes into greater depth about this on his blog

Conclusions

Planning a wedding was stressful, but well worth the effort. I hope these tips help you plan your future wedding. Enjoy!

Also awhile back I added images to my blog, yay! I haven't gotten around to writing a post about it but hopefully pictures will become more common in my posts now that my life is returning to normal.

Categories: shoutouts updates

Marissa Mayer's Work-From-Home Ban is not a Feminist Issue

We wanted Marissa Mayer to be our heroine, our champion — the woman who had it all. She would be a great mom, a great company leader, and at the same time fix what was broken about corporate policy where it relates to women. It's now obvious that's not going to happen.

Throughout all this, we have started treating every decision she makes as a feminist issue and that's a problem. Banning work-from-home is a work place issue. Yes, the majority of people who work-from-home are mom's who are juggling work and being the main caretaker of their family, but there are plenty of others who are not. The problem with highlighting it as a feminist issue is that it clouds the real problem: the lack of trust and respect Mayer has for ALL her employees. Men and women are both affected by this policy change.

Anyone who has read Daniel Pink's "Drive" knows the studies on how to motivate workers. It's not money or being watched every second, it's autonomy, purpose, and mastery. And when you take away an employee's ability to make a choice about how and when they work, you're taking away their autonomy and their happiness. By disallowing her employees to work from home she is telling them "I don't trust you and I don't care about your autonomy.". That's a bold statement and it will definitely hurt her relationship with everyone at Yahoo.

I understand the sentiment of an employer needing to know everyone is working. The problem is that if she feels that people are taking advantage of remote work and not actually working, making sure she can see them in the office isn't going to fix that. Unmotivated people aren't necessarily motivated by being watched. They will do the bare minimum to appear to be getting work done. If she believes employers are abusing the work-from-home policy, forcing them into the office doesn't mean productivity or creativity will rise.

Full disclosure: I'm somewhat biased, being a remote worker myself. I personally don't do well without quiet working spaces or with office drama. I find that environment distracting and since moving to remote-work my productivity has soared. My stress level has dropped dramatically and part of that is not having to deal with a half-hour commute each way. Being able to control my own environment is key to my creative process.

My company is almost completely comprised of remote workers. The reason is because we wanted the best talent the US had to offer, not the best talent within a 50 mile radius of our headquarters. Remote work isn't for everyone, but it's up to managers to determine who is and who isn't capable to producing work at home. Completely banning remote work will hurt employee moral, and that won't be good for Yahoo.

For a company that's already flailing this is not the way to draw new talent. Yahoo needs to figure out how to attract the top-notch developers that are flocking to Google, Facebook, or various well-funded start-ups. When Silicon Valley has all the choice, why would anyone choose to work for a company that so blatantly states that it doesn't respect their employees time or lives outside of work?

Categories: opinion

2012 Reflections

I cannot believe that 2012 is over. This year has been incredible. I've learned massive amounts about coding and my own life it's hard to comprehend it all.

About 1 year ago I started this blog. It was my first real rails project and I take great pride in it. I've neglected it a bit since changing jobs but now that I've finally settled in to my new life I'll be back. I have a lot of posts geared up to be written, and have recently added paperclip.

Every year when I reflect on what I've learned I amaze myself. And really that's the most important thing; not setting resolutions and keeping them, but rather making sure we are always learning and growing. This time last year I had only known Rails for a few months. I had no idea how much in love with it I'd be or that it would set me on a new life path. I had already been involved in coding, but mostly WordPress sites.

The Rails community is so amazing. I had the joy of attending two Rails conferences this year and both were amazing; but they made me realize how much of a minority I am in my own industry. I don't want to harp too much on things that everyone is already discussing, but I think the community has grown a lot as well over the last year. It's good that we're having conversations about sexism and inclusion, it makes us aware of our own faults and strengths as a whole and individually and I am excited to be a part of this new journey with the tech community.

My favorite person and long time boyfriend asked me to marry him in September and I of course said yes. He has been such an integral part in my personal and career growth and I can't thank him enough for the hours he has spent cheering me on. I wouldn't be where I am today without him. Check out Abe's blog abeuchitelle.com.

I also changed jobs. I decided that I no longer wanted to work on projects I didn't fully believe in. The company I worked for was an amazing part of my growth but it was time for me to fly the nest. It's really hard making decisions like that because you always feel like you're letting someone down. But I needed to do it for my own personal growth and for my career. I wanted to work with Rails and Ruby every day.

It's considerably different going from doing agency work to focusing on one project with one team. I'm enjoying working from home and I have more time to do the things I love because I'm not wasting an hour of my life in the car commuting. I've got to say it seems that time goes so much faster and I have gained a better work/life balance from it.

I hope you all had an incredible year as well and have an even more incredible 2013 It's going to be a good year for tech, and I hope my and your personal lives get better in 2013.

Happy New Year my friends!

Categories: updates

Nifty Methods: cycle(first_value, *values)

Recently I was working on an app and needed to "cycle" through some values. Previously, the code was using each_with_index and using that index to figure out where the "odd" or "even" class was applied to the table rows. Since I needed to change the code to use map I could no longer use each_with_index effectively.

First my thinking was to use modulo to loop through and add the classes, but that seemed to be complete overkill for my situation.

I then turned to google to find something simpler. It seems Rails already has a dead simple solution. It amazes me every time I find something that I don't feel is necessary to write from scratch already in existence.

The method is cycle(first_value, *values). This can be used for many things, but the most obvious is cycling through "even" and "odd" classes, but can also be used to more advanced arrays. The Rails API uses cycling through colors as an example. The method also can accept a Hash to create a named cycle.

If you're interested in reading more about this nifty little method, you can find more details in the Rails API.

The Pros of Working from Home: My First Two Weeks in the "New" Office

I recently changed jobs and now work as a Senior Developer at PhishMe! I love my new job, coworkers and my new "office". Being that the company is based in Virginia and I'm not, I now work from home. When I tell people that I almost always get "Oh, I could never do that...". I used to think the same thing, but now I know (at least for the type of person I am) that working from home is AWESOME.

I don't think working from home is for everyone. You have to make a great environment for yourself to work in and you have to be a self-starter. Understanding how you work and what gets in your way the most is important. For me, human's are a major distraction, I can't concentrate if there are people around me talking or moving. Sense wise, I'm extremely sensitive to my surroundings and if I don't want music playing or people talking it can be detrimental to productivity for me; and I'm completely aware that this is my own neuroticism.

So by this point you're probably wondering what I like most about working from home?

Ability to control my environment

The number one thing I enjoy is the ability to control my environment. I cannot stress how amazing this makes my day. I get to control the temperature in my home office. If it's cold, I turn on the space heater, if I'm hot, I open a window. I can be as cold or as hot as I desire, and changing that quickly is not difficult. There is no longer a "consensus" on temperature required.

Having my own space

This office is mine. I own my space and I'm CEO of my life. In the type of work environments we are moving towards there is very little room for "your own time" or space. I found that in a big office where all the space was shared, that it was hard for me to get "in the zone". I couldn't focus on one problem because I was always being asked (not always literally) to focus on something else whether it be talking, or a meeting, or a new project, or a maintenance task. I could never just mentally hone in on my work.

Food

Oh my, all I keep thinking is that "sad sandwiches" are out of my life. If you don't know what a sad sandwich is, you are very very lucky. I can have hot food whenever I want. For breakfast? Yes. For lunch? Yes. For snack? Yes. No more wondering if someone will steal your fancy yogurt, or thinking "Ugg, I guess I'll use the microwave to avoid eating a sad sandwich". I could, if I wanted to spend my entire lunch break making food and eat at my desk, but that lunch will be so much tastier than re-heated leftovers from the night before that just don't taste right microwaved. If you're thinking "Wow, this lady is picky about food preparation" - yes, yes I am.

Flexibility

This might seem odd, but working in an office with mostly males (I'm female) has at times made me self-conscious. Some days your legs are just sooo tight you want to do some stretches at your desk, but bending over in the middle of the office just doesn't seem ideal. Well, now I can do that and there's no one to make me feel weird about it.

Lack of Morning/Evening Commute

No more travel. I never really commuted that far, ~30 min each way, but that's an entire hour of my day, and hour I could never hope to get back. My new commute is short and sweet. The best part is no one can get in my way. I was always dealing with slow drivers or bad drivers causing unnecessary frustrations.

So there you have it, my reasons why working at home is awesome. If you're thinking "could I work at home?" I would make a list of pros and cons. Also, very important is spending the money to make sure your home office is perfect. I made the darkest least favorite room in our house into a lovely, bright, homey room that I want to actually spend time in. It's very comfortable and I love it.

Categories: tips updates

Life Changes and New Hosting!

Wow, it's been awhile since I have written...anything. I have somewhat maybe good excuses. I have been obscenely busy and have unfortunately not been able to write any Ruby or Rails. It's very sad, but all of that will change next Monday because I have found a new job working in information security. I'll get to write rails all day long and I'm so excited.

Another life change that is keeping my very occupied is that I am not engaged and wedding planning is consuming the hours I'm not spending working or filling out paperwork for my new job.

My life will soon return to normal and I will be able to write again. Also since I'm leaving my job I could no longer host on our work servers, so i've moved myself to AWS. So far I like it becasue I can control my own servers and play around a bit. 

I feel like although I haven't written blog posts, my yearly goals are still in place and I'm working towards them. This new job will definitely make me better at Rails, JavaScript and MySQL. I have a ruby gem started and hopefully by next week will be able to get back to working on it.

Categories: updates