What are the things I should look out for when designing a file storage system?
I am working on a side project. I may use GCP or AWS for it.
tldr of the system : Create a web ui to process input files and generate a tabular output giving the link to the dashboard .
I have a bunch of files generated , about 2GB in size in total , Account ID is the high level directory and inside each account there 50 log files each having specific content .
I want to create a UI using Angular framework . Using the UI I will upload the files to GCS /AWS S3 . Then the backend should trigger complex backend scripts, written will be written in Python or Java and then output file is generated. Using the output file a dashboard with few images embedded is generated.
In the end the UI should have a table which should indicate status of processing to COMPLETE and in another column show point to url of the dashboard.
I want to deal with least amount of security work as I want to avoid studying Oauth 2.0 authentication .
Few additional questions , non-system design questions =>
- What are the things I should learn in Angular to cover most bases in this project, I am a backend engineer and going over youtube videos . I want to stick with Angular as I have few work projects going to come in coming months so it will help me to learn it .
- I am unfamiliar with typescript . How much effort will it need to learn in order to code for this project. I am most familiar with Python only.
Discussion
(2 comments)- 1a year agoRobinhood, Meta, Course Hero, PayPal
I'm trying to envision this system: Is it effectively Dropbox? I think this could be a good project to improve raw skills, but I doubt it would get many users as the competition here is fierce. The project also seems pretty complex (rife with edge cases and is full-stack). I recommend checking out our masterclass on coming up with powerful project ideas (both feasible and with potential to get 100k+ users).
In terms of what to look out for:
- Very large file inputs
- Poor network connection (how to pause and resume download based on conditions)
- User exiting the UI mid-upload
- Handling files with same name going into the same directory
What are the things I should learn in Angular to cover most bases in this project, I am a backend engineer and going over youtube videos . I want to stick with Angular as I have few work projects going to come in coming months so it will help me to learn it .
There was a Taro Premium member who volunteered to help with Angular in the Slack community - I recommend reaching out to them.
Zooming out, you might want to just build a pure front-end project to practice Angular, especially if you have work projects coming up for it. Adding the back-end angle will dilute the Angular learnings.
I am unfamiliar with typescript . How much effort will it need to learn in order to code for this project. I am most familiar with Python only.
Probably a lot, but it depends on how you learn. I recommend just jumping in and building things and not spending too much time on tutorials.
- 0a year agoMeta, Pinterest, Kosei
My initial reaction is that this is a very involved side project with lots of moving pieces. If you can get it done, kudos to you! But in my experience doing something this ambitious (you have questions around backend, front-end, and security considerations), I've always failed to make meaningful progress.
I wonder if you can segment the project into phases, or just de-scope parts of it like Alex suggested.
If you're doing anything with JavaScript, please use TypeScript! It just makes life so much easier and productive to have types when you write JS code. There's a bit of a tax for initial setup, but I think you earn that back pretty quickly.
