Parallelized web scraping using RollingCurl
The web is full of information! Your web sites probably already use many APIs for maps, Twitter, IP geolocation, and more. But what about data that's on the web, but doesn't have a readily available API?
The web is full of information! Your web sites probably already use many APIs for maps, Twitter, IP geolocation, and more. But what about data that's on the web, but doesn't have a readily available API?
As a good Drupal developer, one of your New Year's resolutions should be to learn more PHP features. Today, we'll talk about iterating over tree-structured data using the awkwardly-named class RecursiveIteratorIterator.
Drupal allow you to set up installation profiles to fast-track creating a website. Rather than starting from scratch each time you create a site, you can select an install profile that does some initial configuration for you. This is super useful if you make a lot of websites that start the same way. I think multilingual websites are a good example, since there's a lot of configuration that gets repeated.
Often, organizations have big plans for multilingual Drupal. A website might be launched with only one or two languages in place, but the infrastructure is there to add more languages as soon as the translations are ready. Even if you already have two languages in place, there are many steps to adding an additional language.
Drupal 7 gives us the option to include an 'alt text' for each image field. The alt text is used by screen readers or when the image file isn't available. For some organizations with lots of authors, it's hard to get everyone actually using this alt text field. So, sometimes you want to make it required.
I frequently administer remote servers over SSH, and need to copy data to my clipboard. If the text I want to copy all fits on one screen, then I simply select it with my mouse and press CMD-C, which asks relies on my terminal emulator (xterm2) to throw it to the clipboard.
If you have a Drupal multilingual website, you've probably heard about the Entity Translation module, a module that allows you to translate fields in Drupal, instead of translating nodes. This means that instead of creating a node, taxonomy term, and user for each language on your site, you can create a single entity and translate the fields on it. The Entity Translation module includes a submodule that helps you upgrade to this new translation method.
The Web Experience Toolkit distribution of Drupal (aka WxT-Drupal, wetkit, or wet-boew) is a version of Drupal designed for organizations with a bilingual and accessibility requirements. Specifically, it's designed for Canadian government and public organizations. It has built with support for English and French, and includes a theme that provides accessibility and responsive support.
Recently we needed to import a fairly large dataset into Drupal. There were 200,000 items to import, the data changed relatively frequently, and we needed them to be Drupal entities so that we could make them searchable with Apache Solr. We would normally import data using Feeds or Migrate. This time, however, we had higher speed requirements, so we tried something different: importing directly into the database.
Entities in Drupal 7 simplify and centralize common logic that, in earlier versions of Drupal, was duplicated in several places, or only usable in specific modules. There are, however, many features (such as saving entities!) that did not make it into Drupal 7. The Entity API module fills in these missing gaps in the core and goes a step farther by simplifying definition of new entity types, and integration with contrib modules such as Views, Rules, and Features.
Today is not your lucky day. The production server went down due to a hard disk error on the VM's host machine. You don't have a high availability setup because running additional servers 24x7 is quite costly. However, you need a new machine quickly.
Multilingual websites are complicated, due to the wide variety of ways to convey multilingual content to users. Each multilingual website seems to come with a different set of requirements for how content translation is handled. In this post, I'll take a look at approaches to translating content in Drupal 7 and the new possibilities that the Entity Translation module provides.
Whether we like it or not, testing takes a huge amount of time. Often a third of each web development project is spent testing, but it's not usually something that gets a lot of attention.
We've been hearing a lot about responsive design and the future of the web recently. At DrupalCamp Montreal this September, Jen Simmons and Jake Strawn talked about why responsive design is taking hold. I was inspired to try out some of the techniques they talked about, and decided to convert evolvingweb.com to be more responsive.
In Drupal 6, module developers often use Nodes as an all-purpose method for having objects that can be saved into the database. This hack avoids having to write a lot of boilerplate CRUD for all custom data used in your modules. Nodes in D6 are a powerful tool for this use case when combined with the flexibility of CCK's custom fields, and the simplified integration with contrib modules. However, using Nodes for every piece of specialized content means a lot of overhead; contributed modules often modify Nodes and added hooks that operated on them. This is a major performance concern, not to mention a bad architecture for unrelated data.