Quickly create large files in python (fast disk space allocation)

Quickly create large files in python (fast disk space allocation)

Sometimes you want quickly allocate some disk space to a certain file, without worrying about what the content is. Here are two ways to do it in python (in this case I just want to test whether os.stat handles large files correctly, and it turns out to be fine):

Double click to toggle code

import os

with open("newfile", "wb") as f:
    f.seek(1024 * 1024 * 1024 * 40 - 1)
    f.write(b"\0")
os.stat("newfile").st_size

with open("newfile1", "wb") as out:
    out.truncate(1024 * 1024 * 1024 * 40)
os.stat("newfile1").st_size

In some file systems, the first method (seek and write) may give you file appearing to be 40G, but actually only occupying 1 byte on disk, this is called a sparse file system.

Read More

Python module structure example

Python module structure example

Here is the directory tree:

directory tree of the example module

Double click to toggle code

checkbed
├── __init__.py
├── test1.py
└── test2.py

1 directory, 6 files

__init__.py is just an empty files that tells python that this folder can be used as a module. The content of the rest 2 files are listed here:

test1.py

Double click to toggle code

import os
from .test2 import sayhi

class sysutil():
    @staticmethod
    def modpath():
        return os.path.dirname(os.path.realpath(__file__))
    @staticmethod
    def cwd():
        return os.getcwd()
    @staticmethod
    def newsayhi():
        print("Indeed:")
        sayhi()

test2.py

Double click to toggle code

def sayhi():
    print("I said hi.")

Notice how test1.py imports a function from test2.py, by putting a dot before (meaning the current directory), this is called intra-package reference.

Read More

Extra wifi access point with a second router

1. Make sure that internet is working on the first router / IAD / modem
2. Set up wifi SSID and password in the second router, make sure you can access it from your mobile device
3. Disable DHCP on the second router
4. Connect one the LAN ports of the first router to one of the LAN ports of the second
5. Enjoy!

Read More

ubuntu packages to install and settings

ubuntu packages to install and settings

ubuntu packages to install and settings

# math equation editor
equalx

# chinese input
fcitx

# tup the build system for c++
sudo apt-get install libfuse-dev
download and follow instructions.

# armadillo, c++ linear algebra library
# download, unzip, cd into dir
sudo aptitude install liblapack3 liblapack3gf liblapack-dev liblapacke liblapacke-dev liblapack-pic libatlas3-base libatlas3gf-base libatlas-base-dev libatlas-cpp-0.6-1 libatlas-cpp-0.6-dev libatlas-dev libarpack++2c2a libarpack++2-dev libopenblas-base libopenblas-dev cmake
cmake .
make
sudo make install


# zip applications
sudo apt-get install unace unrar zip unzip p7zip-full p7zip-rar sharutils rar uudeview mpack lha arj cabextract

# needed by deepin-scrot
sudo aptitude install python-xlib 


# play dvd
sudo aptitude install libdvdread4 libavcodec-extra
sudo /usr/share/doc/libdvdread4/install-css.sh


# clear type font rendering
infinality

# commandline email:
mutt

# gtk theme engines / icons
gtk2-engines-murrine
gtk2-engines-pixbuf
gtk2-engines-qtcurve
moka

# code tidy up
astyle

# take photos from webcam
cheese

# kde dolphin, thumbnails for video files
kffmpegthumbnailer

# print to pdf
cups-pdf

# newline convertion
dos2unix

# some r packages need this
libcurl4-openssl-dev

# clang c/c++ complier
libclang
clang

lockfile

# mp3 tag editor
puddletag

# screen shooter
scrot

qt4-qtconfig
realpath
gpicview
sage
trash-cli

# clipboard manager on commandline
xsel

# keyboard and mouse automation
xdotool
xvkbd

# kde settings
information tips: no
session management: start with empty session
services: remove unnecessary ones
session start script: ~/bin/startup.sh
font: msyh 10 regular
gtk style: qtcurve
custom shortcuts: import ~/.kaiyin_custom_kde_shortcuts.khotkeys and
global shortcuts: set everything to none, then import  ~/.kaiyin_global_kde_shortcuts.kksrc

# eclipse plugins and settings
# color theme / syntax highlighting
http://marketplace.eclipse.org/content/eclipse-color-theme#.U4yMP3IvC01
# ui theme
http://marketplace.eclipse.org/content/jeeeyuls-eclipse-themes#.U4yM0XIvC00
# vim simulator: vrapper
# vim style ctrl+n ctrl+p selecting autocompletion suggestions: in pref--general--keys, configure "line down" and "line up", in context of "text editing"
# file explorer view: RSE plugin, in marketplace, search for Remote system explorer

# qtcreator settings:
# in fakevim, check pass control keys. if you need any control key in fakevim, just disable it in the normal editor (settings--environment--keyboard)


# things to backup before reinstall:
# kde custom shortcuts
# qtcreator shortcuts
# kde color themes

Read More

Pydev test, fairly good, but not perfect

Pydev test, fairly good, but not perfect

I am trying out the pydev plugin for eclipse, so far the code completion looks ok, but the python console is not reliable at all. Here is a very simple test:

Double click to toggle code

#!/usr/bin/python3

from subprocess import Popen
from subprocess import PIPE
shell_cmd = ["xsel", "-b"]
p = Popen(shell_cmd, stdin=PIPE)
p.communicate("what the hell is this?".encode("ascii"))

This code is just trying to put some text into the system clipboard, and it runs ok in ipython3, yet results are weird in eclipse. The same occurs to pycharm, I have no clue why this is happening.

Read More

Optimal Stopping and Applications

Optimal Stopping and Applications Thomas S. Ferguson Mathematics Department, UCLA

Read More

Qt vs. Java A Comparison of Qt and Java for Large-Scale, Industrial-Strength GUI Development

Qt vs. Java A Comparison of Qt and Java for Large- Scale, Industrial-Strength GUI Development Matthias Kalle Dalheimer

Read More

Genotype collapse function using python

	HB	NA	HE	HA		Genotype	Letter Code	Number Code	PLINK Code
HB	00					Homozygote B	HB	02	00
NA	00	01				NA	NA	NA	01
HE	00	01	00			heterozygote	HE	01	10
HA	00	01	11	11		Homozygote A	HA	00	11
	02	NA	01	00
02	00
NA	00	01
01	00	01	00
00	00	01	11	11
	00	01	10	11
00	00
01	00	01
10	00	01	00
11	00	01	11	11

def collapse(x, y):
    if x == "00" or y == "00":
        return "00"
    elif x == "01" or y == "01":
        return "01"
    elif x == "11" or y == "11":
        return "11"
    elif x == "10" and y == "10":
        return "00"
    else:
        raise Exception("Nonsense combination!")

# test collapse function
allfour = ["00", "01", "10", "11"]
for i in allfour:
    for j in allfour:
        res = collapse(i, j)
        print(res, end="  ")
    print()


# from random import randint
# xint = randint(0, 256)
# yint = randint(0, 256)
def collbyte(xbyte, ybyte):
    x = "{:08b}".format(xbyte)
    xstring = [x[i:i+2] for i in range(0, len(x), 2)]
    y = "{:08b}".format(ybyte)
    ystring = [y[i:i+2] for i in range(0, len(y), 2)]
    collstring = [collapse(xstring[i], ystring[i]) for i in range(4)]
    # fh.write("\n---------------------------------------------\n")
    # fh.write("byte1: {}, binString1: {}".format(xbyte, x) + "\n")
    # fh.write("byte2: {}, binString2: {}".format(ybyte, y) + "\n")
    # fh.write(str(xstring) + "\n")
    # fh.write(str(ystring) + "\n")
    # fh.write(str(collstring) + "\n")
    resbyte = int("".join(collstring), 2)
    return resbyte

import numpy as np
collgen = np.zeros((256, 256), dtype=np.unsignedinteger)
with open("/tmp/collgen.csv", "w") as fh:
    for i in range(256):
        for j in range(256):
            fh.write(str(collbyte(i, j)) + ", ")
        fh.write("\n")

np.savetxt("/tmp/collgen.csv", collgen, delimiter=",")

# def checkbytes(xbyte, ybyte):
#     if collbyte(xbyte, ybyte) != collgen[xbyte][ybyte]:
#         print("Error!")
#
# for i in range(256):
#     for j in range(256):
#         checkbytes(i, j)

Read More

Derivative of cross product

Read More