Posts

Learning Language Model from Scratch Code and Parameters

 So I took Sebastian's large language model code as is.  Note, copyright to him.  But I was curious what are parameters? What are parameters?  See my previous post but... # This is a large model, reducing #LLAMA2_CONFIG_7B = { # "vocab_size": 32000, # Vocabulary size # "context_length": 4096, # Context length # "emb_dim": 4096, # Embedding dimension # "n_heads": 32, # Number of attention heads # "n_layers": 32, # Number of layers # "hidden_dim": 11008, # NEW: Size of the intermediate dimension in FeedForward # "dtype": torch.bfloat16 # NEW: Lower-precision dtype to save memory #} # Previous run # huggingface_hub version: 0.26.1 # sentencepiece version: 0.2.0 # torch version: 2.4.1 # Total number of parameters: 6,738,415,616 # float32 (PyTorch default): 52.33 GB # bfloat16: 26.17 GB LLAMA2_CONFIG_7B = { "vocab_size" : 32000 , # Keeping

Calculating Parameters with LLM

 And this data from Sebastian's book on large language models 124 Million parameters GPT_CONFIG_124M = {     "vocab_size": 50257,     # Vocabulary size     "context_length": 1024,  # Context length     "emb_dim": 768,          # Embedding dimension     "n_heads": 12,           # Number of attention heads     "n_layers": 12,          # Number of layers     "drop_rate": 0.1,        # Dropout rate     "qkv_bias": False        # Query-Key-Value bias } The 1.5 billion parameter GPT model config  GPT_CONFIG_1558M = {     "vocab_size": 50257,     # Vocabulary size     "context_length": 1024,  # Context length     "emb_dim": 1600,         # Embedding dimension  (Change here)     "n_heads": 25,           # Number of attention heads     "n_layers": 48,          # Number of layers     "drop_rate": 0.1,        # Dropout rate     "qkv_bias": False        #

Review of JVM Notebook - JVM Languages

 It has been a while but there are still relevant JVM projects here. See the JVM Notebook.. https://github.com/berlinbrown/jvmnotebook/tree/master/jvmnotebook The Java Virtual Machine ( Sun's JVM is called HotSpot ) is a java bytecode interpreter which is fast, portable and secure. Jython, JRuby, Scala, ABCL (Common Lisp) are popular language implementations that run on the JVM that allow for the jvmcookbook sugar of their particular languages. Review of languages * http://jruby.codehaus.org/ - JRuby Home * http://groovy.codehaus.org/ - Groovy Home * http://clojure.org/ - Clojure * http://www.scala-lang.org/ - Scala is a general purpose programming language designed to express common programming patterns in a concise, elegant, and type-safe way. * http://www.jython.org - Jython * http://common-lisp.net/project/armedbear/ - Armed Bear Common Lisp (ABCL) is an implementation of ANSI Common Lisp that runs in a Java virtual machine. * http://sis

Web Security is Important - Basic SQL Injection Project

 Here is a basic SQL Project Test https://github.com/berlinbrown/zri-banking-forum-injection A SQL injection attack consists of insertion or “injection” of a SQL query via the input data from the client to the application. A successful SQL injection exploit can read sensitive data from the database, modify database data (Insert/Update/Delete), execute administration operations on the database (such as shutdown the DBMS), recover the content of a given file present on the DBMS file system and in some cases issue commands to the operating system.  SQL is a standard language for accessing and manipulating databases. SQL lets you access and manipulate databases.  A database is an organized collection of structured information, or data, typically stored electronically in a computer system.  Databases are used in all types of modern applications including for banking, shopping and more.

Some youtube - vids Atlanta and ML

  https://www.youtube.com/watch?v=bIogyR3aPjs https://www.youtube.com/watch?v=Dg6LMAUBbZY

More Bitcoin related linked

 Bitcoin   https://github.com/bitcoinbook/bitcoinbook https://github.com/dvf/blockchain https://github.com/protofire/blockchain-learning-path https://github.com/ndrwnaguib/napster-filesharing-system/blob/master/peer/downloads/downloaded_1.txt https://github.com/ndrwnaguib/napster-filesharing-system https://github.com/yjjnls/awesome-blockchain https://github.com/openblockchains/awesome-blockchains

My post on raygun

 #Raygun thank you for failing.  It is OK to fail.  Fail, move on.  Our own President Biden failed recently.  Shane Gillis failed on SNL.  George Santos failed.  Ingrid Andress on national anthem.  Doing good is better, but OK to fail.  Who else?