What language should I learn/use with this project?

[Scarpozzi]

Requirements
1. I'm needing to setup, preferably, a web-based system that accepts large CSV files (file upload)
2. The CSV files will need to be checked somehow for formatting errors
3. The files will then need to be processed/filtered/sorted...certain columns or rows counted/summed

After that, data from the CSV file will be pulled into a number of columns. The number of columns can vary, depending on the CSV imported. I'll need to run mathematical equations on the data and ultimately display the results in the web interface and/or export the results in CSV format.

I've done a lot of this stuff in Powershell previously and it would work for what I'm needing, all except for the web front-end. I considered even using mySQL and php, but not sure I want to go that route due to security concerns. Any suggestions?

 

[sourceninja]

This sounds like a perfect job for AWS lambda

 

[Scarpozzi]

This sounds like a perfect job for AWS lambda

Thanks for the reply. I'll check it out tomorrow.

 

[beginner99]

I woudl try python and with that numpy and pandas.

Since you could run it on powershell on your local pc I doubt going into the cloud is needed or even reasonable.

 

[purbeast0]

I woudl try python and with that numpy and pandas.

Since you could run it on powershell on your local pc I doubt going into the cloud is needed or even reasonable.

While I agree that there is no need for cloud computing, he did say he needs to upload a file and do "stuff" to it, so I'm assuming he needs a server and a client.

I'd just go straight up javascript for this, client and server. That's a pretty basic task.

 

[sourceninja]

1. I'm needing to setup, preferably, a web-based system that accepts large CSV files (file upload) - Web based, so probably hosted
2. The CSV files will need to be checked somehow for formatting errors - Simple task, less than 5 minutes run time
3. The files will then need to be processed/filtered/sorted...certain columns or rows counted/summed - Simple task, most likely also under 5 minutes
4. After that, data from the CSV file will be pulled into a number of columns. The number of columns can vary, depending on the CSV imported. I'll need to run mathematical equations on the data and ultimately display the results in the web interface and/or export the results in CSV format.

AWS lamba and API gateway, small static site in front. Could be done in no time using python or javascript and hosted on lamba. Pay only for the runtime, this could cost literally pennies a month. I'd have the lambda function check the do all the heavy work and store the resulting data on S3 and return a signed URL to the client side javascript to display or download to the user. MVP on this is maybe 2 hours.

 

[beginner99]

While I agree that there is no need for cloud computing, he did say he needs to upload a file and do "stuff" to it, so I'm assuming he needs a server and a client.

I'd just go straight up javascript for this, client and server. That's a pretty basic task.

for the website /client yeah js is needed but server side for the actually processing I would call that a terrible idea.

Another option would be KNIME which can do such data manipulation stuff easily. No programming needed. Workflows can be run in batch mode for example: user uploads data to server, server calls workflow in batch mode with the file location as parameter.

Or you could go for the commercial version which includes a web portal to run workflows.

For most here that probably never heard of KNIME, it's the leading data science and machine learning tool in the Gartner magic quadrant anaylsis:

https://www.kdnuggets.com/2018/02/gartner-2018-mq-data-science-machine-learning-changes.html

We use it at work heavily and I would say it is THE TOOL for automation tasks that involve data manipulations.

 

[purbeast0]

Just because you don't like node.js (per the tech thread) doesn't mean it's a terrible idea. This is a very basic task that can be done in tons of different ways, as we've seen from the suggestions in this thread.

 

[Merad]

Language (probably) doesn't really matter for this. Whatever you know best and/or are more comfortable with. Unless the data is very small, I would lean towards putting it in a database of some kind for easy querying/analysis. SQLite may be a good candidate.

 

[beginner99]

Just because you don't like node.js (per the tech thread) doesn't mean it's a terrible idea. This is a very basic task that can be done in tons of different ways, as we've seen from the suggestions in this thread.

True. i can also use a hammer to put in a screw.. It's just a lot harder than using a screw-driver.

 

[Scarpozzi]

I can start local with python. Uploading the file to a web is a requirement as/if the project scales. Sounds like lambda is a gtgre opportunity.

I've been slammed. I've got to teach myself python and get started if I can design the right data structures and/or steal some functions to make this work.

Thanks.

 

[mxnerd]

Take a look at CSVKit in Python

https://csvkit.readthedocs.io/en/1.0.3/

 

[beginner99]

I can start local with python. Uploading the file to a web is a requirement as/if the project scales. Sounds like lambda is a gtgre opportunity.

I've been slammed. I've got to teach myself python and get started if I can design the right data structures and/or steal some functions to make this work.

Thanks.

again, look at pandas (and numpy).

 

[Scarpozzi]

again, look at pandas (and numpy).

Numpy seems to be what I need. Going to check pandas. I actually nudged my best friend/programming buddy of mine and he may be working on this now.

 

[sao123]

Use the language you are most comfortable with.
I would do this in asp.net (c#) with Sql server express and be done in 3 days.

 

[Scarpozzi]

Use the language you are most comfortable with.
I would do this in asp.net (c#) with Sql server express and be done in 3 days.

My partner in crime on this project spent 2 hours working on it before I had time (because my kids are younger than his) and got the bulk of the work done in 16 lines of code using python, numpy, and pandas. The next day, he worked some more and bulked the code up to about 120 lines and is pretty much done. Thanks for the info. I passed it on to him and he was able to knock it out. I'm going to post another thread for my next question because it's more general toward drupal.

 

[Bradtech519]

I used ruby to manipulate large text files, format content inside it so they are readable by the resolv gem, and perform 100s of DNS queries externally based on the hostnames sent in a txt file which are dynamically generated & funky formatted. Found it very powerful & many gems that helped. Runs great on Linux or Windows with file paths as parameters that are input from the users. I'm sure python or go can do the same. As can powershell/.net. If you have plenty of time to get this done(I didn't). Look into something you don't know well and learn from the experience.