• 2 Posts
  • 30 Comments
Joined 6 months ago
cake
Cake day: January 3rd, 2024

help-circle

  • It was a mixture of factors.

    Data was to be dumped into a S3 bucket (minio), this created an event and anouther team had built an orchestrator which would do a couple of things but eventually supply an endpoint a reference to a plain/txt file for analysis.

    For the Java devs they had to [modify the example camel docs.](https://camel.apache.org/manual/rest-dsl.html) and use the built in jackson library to convert the incoming object to a class. This used the default AWS S3 api to create a stream handle which fed into the OpenNLP docs. .

    The Python project first hit a wall in setting up Flask. They followed the instructions and it didn’t work from setup tools. The Java team had just created a new maven project from the Intelij but the same approach didn’t work for the Python team using pycharm. It lost them a couple days, I helped them overcome it.

    Then they hit a wall with Boto3, the team expected to stream data but Boto3 only supports downloading, there was also a complexity issue the AWS SDK in Java waa about 20 lines to setup and a single line to call, it was about 50 lines in Python. On the positive side I got to explain what all the config meant in S3.

    This caused the team anouther few days of delay because the team knew I used a 350MiB Samsung TV guide to test the robustness and had to go learn about Docker volume mounting and they thought they needed a stateful kubernetes service and I had to explain why that was wrong.

    Basically Python threw up a lot of additional complexity and the docs weren’t as helpful as they could have been.



  • You do, but considering the scales they process data I suspect Google would be better building Go tooling (or whatever the dominate internal language is).

    A few years back I was trying to teach some graduates the importance of looking at a programming language ecosystem and selecting it based on that.

    One of my comparison projects was Apache OpenNLP/Camel vs Flask/Spacy.

    Spacy is the go to for NLP, I expected it to be either quicker to develop, easier to use, better results or just less resources.

    I assigned Grads with Java experience Spacy and Python experience OpenNLP.

    The OpenNLP guys were done first, they raved about being able to stream data into the model and how much simplier it made life.

    When compared with the same corpus (Books, Team emails, corporate sharepoint, dev docs, etc…) OpenNLP would complete on 4GiB of RAM in less than a second on 0.5vCPU. Spacy needed 12GiB and was taking ~2 seconds with 2vCPU. They identified the same results…

    Me and a few others ended up spending a day reading the python and trying to optimise it, clearly the juniors had done something daft, they had not.

    It rather undermined my point.


  • My expectation is whatever the solution it needs to dockerise and be really easy to deploy via docker compose or Kubernetes so people can quickly and easily set up their own.

    The front end is effectively static files so I would probably choose Apache or Express (whichever gives me a smaller docker image)…

    For the backend I would choose Java for Spring Boot. An Alpine image with OpenJDK and the app is tiny. Spring has a library for every kind of interface making them trivial to implement but the main reason is hibernate.

    Hibernate (now Spring Data) was the first library for being able to switch out databases without having to change code (its all config). A lot of mastodon instances struggle with the resource requirements of elastic search so letting small instances use something like postgres would seem ideal.

    I have noticed Go/Rust still expect you to write or manage a lot of stuff Spring gives away for free. Python is ok if your backend is really tiny but there is a lot of boilerplate in how Python libraries work so complex projects get hard to manage and I assume interacting with the fediverse will add complexity.






  • When AMD launched Ryzen they deliberately offered way more I/O bandwidth than Intel.

    The first generation Ryzen CPU’s used RAM frequency that could cause performance issues if you used low frequency RAM. That got fixed in the 3000 series.

    There are a small number of Ryzen CPU’s which end with “3D,” it means they had 3D Cache memory and its supposed to add rediculous performance in certain situations. Phoronix runs tons of benchmarks on CPU and GPU.

    The only Intel instructions AMD haven’t implemented is AVX-512 and AVX-10. No one uses AVX-512 as Intel CPU’s get so hot they performance throttle so much its faster to not use the extension. AVX-10 is something new Intel released this year to get around that.

    AMD does support AVX2 which a lot of Audio/Video products do use.


  • You seem to be intentionally missing the point, but to reiterate…

    You shower before entering a pool to wash the dirt from your body off (your cleaning yourself).

    The more of your body covered the less effective that shower is.

    Ideally everyone would be naked in the shower, but there are probably outfits which increasingly render the shower less and less effective (e.g. speedos are better than shorts, etc .).

    It would not surprise me if a Burkina covered so much that the cleaning shower is rendered pointless


  • The shower before a pool is to ensure people aren’t entering the pool coated in dirt (e.g. sweat, hair, dead skin, etc…).

    The chemicals in a pool are designed to bind to that dirt and kill any bacteria introduced.

    There is a limit to the chemicals you can add to a pool (before it hurts humans) and once the amount has activated you need to drain the pool and refill it.

    Swimming pools hold crazy amounts of water which is also really expensive to heat up, so pools want to do that as little as possible.

    Clothing interfers with cleaning your body, so people entering near fully clothed (e.g. like a Burkina) will likely introduce more dirt into the pool.

    That translates into increased costs for swimming pools or pools which maintain the old schedule and just operate unsafely.

    This is all based on owning a hot tub and learning how to maintain it.

    Hopefully this also explains why it doesn’t matter people enter the sea fully clothed




  • Firstly it was just a bit of fun but from memory…

    Twitter was listed as having 2 data centers and a couple dozen satellite offices.

    I forgot the data center estimate, but most of those satelites were tiny. Google gave me the floor area for a couple and they were for 20-60 people (assuming a desk consumes 6m2 and dividing the office area by that).

    Assuming an IT department of 20 for such an office is rediculous but I was trying to overestimate.


  • The Silicon Valley companies massively over hired.

    Using twitter as an example, they used to publicly disclose every site and their entire tech stack.

    I have to write proposals and estimates and when Elon decided to axe half the company of 8000 I was curious…

    I assigned the biggest functional team I could (e.g. just create units of 10 and plan for 2 teams to compete on everything). I assumed a full 20 person IT department at every site, etc… Then I added 20% to my total and then 20% again for management.

    I came up with an organisation of ~1200, Twitter was at 8000.

    I had excluded content moderators and ad sellers because I had no experience in estimating that but it gives a idea of the problem.

    I think the idea was to deny competition people but in reality that kind of staff bloat will hurt the big companies


  • It does but for the 90’s/00’s a computer typically meant Windows.

    The ops staff would all be ‘Microsoft Certified Engineers’, the project managers had heard of Microsoft FuD about open source and every graduate would have been taught programming via Visual Studio.

    Then you have regulatory hurdles, for example in 2010 I was working on an ‘embedded’ platform on a first generation Intel Atom platform. Due to power constraints I suggested we use Linux. It worked brilliantly.

    Government regulations required anti virus from an approved list and an OS that had been accredited by a specific body.

    The only accredited OS’s were Windows and the approved Anti Viruses only supported Windows. Which is how I got to spend 3 months learning how to cut XP embedded down to nothing.



  • Technical Leads are not rational beings and lots of software is developed from an emotional stand point.

    Engineering is trade offs, every technical decision you make has a pro/con.

    What you should do is write out the core requirements/constraints.Then you weigh the choices to select the option that best meets it.

    What actually happens is someone really likes X framework, Y programming language or Z methodology and so decides the solution and then looks for reasons to justify it.

    Currently the obvious tell is if they pitch Rust. I am not saying Rust is bad, but you’ll notice they will extoll the memory safety or performance and forget about the actual requirements of the project.



  • There will always be someone who is beating you in a metric (buying houses, having kids, promotions, pay, relationships, etc…) fixating on it will drive you mad.

    Instead you should compare your current status against where you were and appreciate how you are moving forward

    As for age

    During university my best mate was 27 who dropped out of his final year, grabbed a random job, then went to college to get a BTEC so they could start the degree.

    It was similar in my graduate intake, we had a 26 year old who had been a brickie for 5 years before getting a comp sci degree.

    The first person I line managed was a junior 15 years older than me, who had a completely different career stream. They had the house, kids, had managed big teams, etc… honestly I learnt tons from them.


  • It isn’t a good move.

    A domain name can cost as little as £10, similarly most email services cost ~£5-£15 per person per month. Its normally pretty easy to link a domain to an email provider and doesn’t cost anything other than time.

    If a company can’t be bothered to implement the most basic online branding people will make their assumptions and some will filter your company out because of it. With the cost to implement so low (e.g. £160 per year), even the loss/gain of a single customer would justify it.