Archive

Archive for December, 2011

Machine Learning Blogs

December 31, 2011 Leave a comment

Here’s the OPML for some good machine learning blogs. Well this Q&A posts a lot others
Maybe I should continue collecting them 🙂

Random Links for AWS, MapReduce, rpy2

December 29, 2011 Leave a comment

Still having the issue using rpy2 in userdata sent to EC2 instance. Everything works well when opening the terminal of the instance created by my own image, but fails when sending userdata via boto. Tried several testing for test the rpy2, like python -m ‘rpy2.tests’, went through well, but still not work.
wired error like:

RRuntimeError”, “<no args> \”/usr/local/lib/python2.7/dist-packages/rpy2-2.2.4dev_20111122-py2.7-linux-x86_64.egg/rpy2/robjects/__init__.py\”, line 225, in __call__\n res = self.eval(p)\n”, ” File \”/usr/local/lib/python2.7/dist-packages/rpy2-2.2.4dev_20111122-py2.7-linux-x86_64.egg/rpy2/robjects/functions.py\”, line 82, in __call__\n return super(SignatureTranslatedFunction, self).__call__(*args, **kwargs)\n”, ” File \”/usr/local/lib/python2.7/dist-packages/rpy2-2.2.4dev_20111122-py2.7-linux-x86_64.egg/rpy2/robjects/functions.py\”, line 34, in __call__\n res = super(Function, self).__call__(*new_args, **new_kwargs)\n”]]

Below just some random links for records.

Amazon Web Service: http://aws.amazon.com/elasticmapreduce/
Get started with Amazon EC2 by using using the boto library for Python: http://aws.amazon.com/articles/3998?_encoding=UTF8&jiveRedirect=1
Documentation of Boto, online at http://boto.cloudhackers.com

Cloud vs AMR:
http://umichcloud.blogspot.com/2011/11/gui-and-running-mapreduce-from-desktop.html
Best Cloud Apps and Service: https://sites.google.com/site/truthkos/old-pages/ci-days-research-in-the-cloud/favoritecloudapps

Use R in Amazon MapReduce
http://blog.revolutionanalytics.com/2009/05/running-r-in-the-cloud-with-amazon-ec2.html
http://benreuven.com/udiwiki/index.php?title=R_with_Amazon_MapReduce

Create Amazon Machine Image:
http://docs.amazonwebservices.com/AWSEC2/2007-08-29/GettingStartedGuide/creating-an-image.html
http://docs.amazonwebservices.com/AmazonEC2/gsg/2006-10-01/
http://ged.msu.edu/angus/tutorials/creating-custom-amis.html

Lunch the instance and save the key-value pair:
http://ged.msu.edu/angus/tutorials/renting-a-computer-from-amazon.html

R-glm
http://data.princeton.edu/R/glms.html

Chain-MapR-Job
http://blog.data-miners.com/2008/02/mapreduce-and-k-means-clustering.html

http://stackoverflow.com/questions/2499585/chaining-multiple-mapreduce-jobs-in-hadoop
http://stackoverflow.com/questions/4170060/hadoop-map-reduce-chaining

http://stackoverflow.com/questions/2986271/need-help-implementing-this-algorithm-with-map-hadoop-mapreduce

http://stackoverflow.com/questions/6446914/implementing-parallel-for-in-hadoop
http://stackoverflow.com/questions/2986271/need-help-implementing-this-algorithm-with-map-hadoop-mapreduce
http://stackoverflow.com/questions/6523911/implementing-cross-join-in-hadoop
http://stackoverflow.com/questions/6800438/parallel-reducing-with-hadoop-mapreduce
http://stackoverflow.com/questions/7009930/shared-variable-in-map-reduce
http://stackoverflow.com/questions/2888788/global-variables-in-hadoop
http://stackoverflow.com/questions/1217850/streaming-data-and-hadoop-not-hadoop-streaming
http://stackoverflow.com/questions/2333618/hadoop-one-map-and-multiple-reduce

http://database.cs.brown.edu/sigmod09/benchmarks-sigmod09.pdf
http://www.econsteve.com/r/barr-parallelPresoFeb2011.pdf

RF tree size

December 29, 2011 Leave a comment

The number of trees in Random Forest is suggested to be high enough in order to ensure that every input sample gets predicted at least a few times. It is suggested that if want auxiliary information like variable importance or proximity , grow a large number of trees is agg choice, since more stable results can be obtained. Well for the current testing data, I have seen that it is not straight proportional to the performance. Features from POS and NEG classes are not containing very good discrimination information, so even a very complex tree can not save the performance.

[Ref] http://oz.berkeley.edu/users/breiman/Using_random_forests_V3.1.pdf

Categories: Data Mining

Name Prefix

December 29, 2011 Leave a comment

Some name prefix links collected before. Many of them are royal titles.

Mr., Mrs., Ms., Miss, Dr., A.V.M., AB, Adm., Amb, AMN, Archbishop, Baron, Baroness, Bishop, Brig. Gen., Brigadier, Bro., Cantor, Capt., Cardinal, Chaplain, Cmdr., CMSGT, Col., Consul, Count, Countess, Cpl., CPO, CWO, Dean, Duchess, Duke, Earl, Ens., Eur Eng, Father, Fr., Gen., Gov., H. E., Herr, Hon., HRH, Lady, Lord, Lt., Lt. Cmdr., Lt. Col., Lt. Gen., M., Maj., Maj. Gen., Master, Mlle., Mme., Mother, MSGT, Pastor, PFC, Pres., Prince, Princess, Prof., Rabbi, Radm., Rev., Rt. Hon., Senator, Sgt., Sgt. Maj., Sir, Sister, SMSGT, Speaker, Squad.Ldr., Sr., SrA, Sra., Srta., SSGT, Swami, TSGT, Vadm.

http://www.flyertalk.com/forum/american-aadvantage/642484-aadvantage-name-prefix-options.html

http://www.morganhorse.com/registry/prefix_lookup/

Categories: NLP

rpy2 error when importing numpy array

December 14, 2011 Leave a comment

Get error when importing arrays from numpy to rpy2:

ValueError: Nothing can be done for the type <type ‘numpy.ndarray’> at the moment.
 Tried adding two additional import commands, and solved the problem:
import rpy2.robjects.numpy2ri
rpy2.robjects.numpy2ri.activate()
Categories: Python, R

Weekly Record

December 13, 2011 Leave a comment

1. Ran into problem of “Task attempt failed to report status for 6003 seconds. Killing!”

Figure out that it is due to eliminating (not output anything) when have missing features. Some part of data could have hugh part as missing feature, which cause the map-reduce status not updating for a long time. Basically, the error means that the task stayed in map or reduce for more then allowed time but with no stdin/stdout.

Changed the code, but there are also other ways to solve this by increasing the timeout parameter. Here’s a link to this
<https://issues.apache.org/jira/browse/MAPREDUCE-1308>
<http://stackoverflow.com/questions/5864589/how-to-fix-task-attempt-201104251139-0295-r-000006-0-failed-to-report-status-fo>

Another way is to use the Reporter.

Reporter is a facility for MapReduce applications to report progress, set application-level status messages and update Counters.

Mapper and Reducer implementations can use the Reporter to report progress or just indicate that they are alive. In scenarios where the application takes a significant amount of time to process individual key/value pairs, this is crucial since the framework might assume that the task has timed-out and kill that task. Another way to avoid this is to set the configuration parameter mapred.task.timeout to a high-enough value (or even set it to zero for no time-outs).

A Java code to check the status:

http://code.google.com/p/appengine-mapreduce/source/browse/trunk/python/src/mapreduce/context.py

cnkiTable.put(put);

if ((++count % 1000)==0) {
context.setStatus((count / len) +” done!”);
context.progress();
}


2. Boto connection error:

The requested instance type’s architecture (i386) does not match the architecture in the manifest for ami-c9c70da0 (x86_64)</Message></Error></Errors><RequestID>b71b1ee4-5a98-45e2-af1d-7da0db114afb

It is saying that my created image is 64bit, while the instance going to lunch is32bit. Tried a test on lunch the instance using a 32bit image. Went through well.

Finally, found out that it is because of instance_type is not correct.

 

3. Other collections on AWS
https://cwiki.apache.org/MAHOUT/mahout-on-elastic-mapreduce.html

Possible to use the customized AMI for Elastic MapReduce on AW: Elastic MapReduce doesn’t support customer AMIs at this time. The service instead has a feature called “Bootstrap Actions” that allows you to pass a reference to a script stored in Amazon S3 and relatedarguments to Elastic MapReduce when creating a job flow. This script is executed on each job flowinstance before the actual job flow runs. This post describes how to create bootstrap actions:
http://developer.amazonwebservices.com/connect/entry.jspa?externalID=3938&categoryID=265 (section “Creating a Job Flow with Bootstrap Actions”)

Processing images is one of the typical Elastic MapReduce use cases.

Use S3 or hdfs:

I would like to be able to access and use HDFS directly instead of having to worry about using the S3 bucket for initial or intermediate IO. I am worried about the IO performance of the S3 bucket against the HDFS performance when accessing the S3 bucket.I have seen multiple posts that say it doesn’t matter and others that say it can matter.

HDFS vs S3 provide different benefits; HDFS has lower latency but S3 has higher durability. In terms of long term storage (without compute) S3 is the cheaper option.

Would people recommend using EMR or EC2 with a Hadoop 0.20 image for doing something like this?

EMR is highly tuned to offer the best performance possible with S3.

Does the EMR setup support using the HDFS like this with custom JARs?

Definitely. Intermediate data is stored in HDFS unless you configure things otherwise. You are able to choose whether to use HDFS or S3 for your initial data.

Common Problems Running Job flows. <https://forums.aws.amazon.com/thread.jspa?threadID=30925>
Using s3:// instead of s3n://

The Amazon Elastic Mapreduce Instances run on s pre-defied AMI. To use the customized Intance for MapReduce, an way is to run a boostrap action
[1]<http://serverfault.com/questions/253917/what-ami-is-launched-on-amazons-elastic-mapreduce-instances>.

[2]< http://aws.typepad.com/aws/2010/04/new-elastic-mapreduce-feature-bootstrap-actions.html>

[3]< http://atbrox.com/2010/10/01/programmatic-deployment-to-elastic-mapreduce-with-boto-and-bootstrap-action/>

[4]< https://github.com/atbrox/atbroxexamples>

Q: What is Amazon Elastic MapReduce Bootstrap Actions?

Bootstrap Actions is a feature in Amazon Elastic MapReduce that provides users a way to run custom set-up prior to the execution of their job flow. Bootstrap Actions can be used to install software or configure instances before running your job flow.

Q: How can I use Bootstrap Actions?

You can write a Bootstrap Action script in any language already installed on the job flow instance including Bash,Perl, Python, Ruby, C++, or Java. There are several pre-defined Bootstrap Actions available. Once the script is written, you need to upload it to Amazon S3 and reference its location when you start a job flow. Please refer to the “Developer’s Guide”: http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/ for details on how to use Bootstrap Actions.

Q: How do I configure Hadoop settings for my job flow?

The Elastic MapReduce default Hadoop configuration is appropriate for most workloads. However, based on your job flow’s specific memory and processing requirements, it may be appropriate to tune these settings. For example, if your job flow tasks are memory-intensive, you may choose to use fewer tasks per core and reduce your job tracker heap size. For this situation, a pre-defined Bootstrap Action is available to configure your job flow on startup. See the Configure Memory Intensive Bootstrap Action in the Developer’s Guide for configuration details and usage instructions. An additional predefined bootstrap action is available that allows you to customize your cluster settings to any value of your choice. See the Configure Hadoop Bootstrap Action in the Developer’s Guide for usage instructions.

http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/

http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/index.html?Bootstrap.html#PredefinedBootstrapActions_ConfigureHadoop

Using AmazonEC2 create the image with hadoop <http://wiki.apache.org/hadoop/AmazonEC2>

Categories: Amazon Web Services

Keep Tring

December 8, 2011 Leave a comment

Always the question, how to perform timely and scalable analytical processing of large datasets.

Let’s start building the house piece by piece.

Categories: Uncategorized Tags: