Wednesday 12 September 2018

Finding duplicate values in MySQL

Do a SELECT with a GROUP BY clause. Let's say name is the column you want to find duplicates in:
SELECT name, COUNT(*) c FROM table GROUP BY name HAVING c > 1;
This will return a result with the name value in the first column, and a count of how many times that value appears in the second

How do I escape ampersands in batch files?

How do I escape ampersands in a batch file (or from the Windows command line) in order to use the start command to open web pages with ampersands in the URL?
Double quotes will not work with start; this starts a new command-line window instead.
Update 1: Wael Dalloul's solution works. In addition, if there are URL encoded characters (e.g. space is encoded as %20) in the URL and it is in a batch file then '%' must be encoded as '%%'. This is not the case in the example.
Example, from the command line (CMD.EXE):
start http://www.google.com/search?client=opera&rls=en&q=escape+ampersand&sourceid=opera&ie=utf-8&oe=utf-8
will result in
http://www.google.com/search?client=opera 
being opened in the default browser and these errors in the command line window:
'rls' is not recognized as an internal or external command,
operable program or batch file.
'q' is not recognized as an internal or external command,
operable program or batch file.
'sourceid' is not recognized as an internal or external command,
operable program or batch file.
'ie' is not recognized as an internal or external command,
operable program or batch file.
'oe' is not recognized as an internal or external command,
operable program or batch file.
Platform: Windows XP 64 bit SP2.



From a cmd:
  • & is escaped like this: ^& (based on @Wael Dalloul's answer)
  • % does not need to be escaped
An example:
start http://www.google.com/search?client=opera^&rls=en^&q=escape+ampersand%20and%20percentage+in+cmd^&sourceid=opera^&ie=utf-8^&oe=utf-8
From a batch file
  • & is escaped like this: ^& (based on @Wael Dalloul's answer)
  • % is escaped like this: %% (based on the OPs update)
An example:
start http://www.google.com/search?client=opera^&rls=en^&q=escape+ampersand%%20and%%20percentage+in+batch+file^&sourceid=opera^&ie=utf-8^&oe=utf-8

Thursday 17 May 2018

Disable SIP ALG in Cisco 1800 Series Router

no ip nat service sip udp port 5060
no ip nat service sip tcp port 5060

Batch file rename with random alphanumeric

The following simple script will rename all .jpg files in the current folder to random 8 character alphanumeric names, preserving the .jpg extension. Note that it is possible for the random name generator to produce a name that already exists, so the script loops until it successfully generates a unique name.
@echo off
setlocal disableDelayedExpansion
set "chars=ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"
for /f "eol=: delims=" %%F in ('dir /b /a-d *.jpg') do call :renameFile "%%F"
exit /b

:renameFile
setlocal enableDelayedExpansion
:retry
set "name="
for /l %%N in (1 1 8) do (
  set /a I=!random!%%36
  for %%I in (!I!) do set "name=!name!!chars:~%%I,1!"
)
echo if exist !name!.jpg goto :retry
endlocal & ren %1 %name%.jpg
A bit more code yields a versatile utility that allows you to specify the source path and file mask, and also provides the option to process sub-directories as well. The utility always preserves the extension of each file. Be careful with this utility!
renameFilesRandom.bat
:: renameFilesRandom.bat  [filter]  [/s]
@echo off
setlocal disableDelayedExpansion

:: Parse and validate arguments
set "option="
set "filter="
if "%~3" neq "" (
  >&2 echo ERROR: Too many arguments
  exit /b 1
)
if /i "%~1" equ "/S" (set "option=/S") else if "%~1" neq "" set "filter=%~1"
if /i "%~2" equ "/S" (set "option=/S") else if "%~2" neq "" (
  if defined filter (
    >&2 echo ERROR: Only one filter allowed
    exit /b 1
  ) else set "filter=%~2"
)
if "%filter:~0,1%" equ "/" (
  >&2 echo ERROR: Invalid option %filter%
  exit /b 1
)
if not defined filter set "filter=*"

:: Convert a directory filter into a file filter with wildcards
if exist "%filter%\" set "filter=%filter%\*"

:: Determine source if /S option not specified
set "src="
if not defined option for /f "eol=: delims=" %%F in ("%filter%") do set "src=%%~dpF"

:: Rename the specified files
set "chars=ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"
for /f "eol=: delims=" %%F in ('dir /a-d /b %option% "%filter%"') do call :renameFile "%%F"
exit /b

:renameFile
setlocal
if not defined src set "src=%~dp1"
set "old=%~nx1"
set "ext=%~x1"
setlocal enableDelayedExpansion
:retry
set "name="
for /l %%N in (1 1 8) do (
  set /a I=!random!%%36
  for %%I in (!I!) do set "name=!name!!chars:~%%I,1!"
)
if exist "!src!!name!!ext!" goto :retry
ren "!src!!old!" "!name!!ext!"
Below are some sample usages.
Rename all files in the current directory:
renameFilesRandom
Rename all .jpg files in the current directory:
renameFilesRandom *.jpg
Rename all .jpg files in the c:\test folder, and all its sub-folders (recursive):
renameFilesRandom c:\test\*.jpg /s

Directory.GetFiles () Order By Date

  1. using System.Linq;
  2.  
  3. DirectoryInfo info = new DirectoryInfo("");
  4. FileInfo[] files = info.GetFiles().OrderBy(=> p.CreationTime).ToArray();
  5. foreach (FileInfo file in files)
  6. {
  7.     // DO Something...
  8. }

How to execute ImageMagick to convert only the first page of the multipage PDF to JPEG?

If you are using a convert command line you can execute it with these parameters:
convert  source.pdf[0]  output.jpeg
Note that the page count of ImageMagick is 0-based. So [0] means 'page 1'. To select, say the 4th page, you'd have to use [3].
This syntax does not only work for PDF input. It also works with other multi-page or mult-frame formats, such as multi-page TIFF or animated multi-frame GIFs and PNGs.

Process.Start Method

using System.Diagnostics;

class Program
{
    static void Main()
    {
        LaunchCommandLineApp();
    }

    /// <summary>
    /// Launch the legacy application with some options set.
    /// </summary>
    static void LaunchCommandLineApp()
    {
        // For the example.
        const string ex1 = "C:\\";
        const string ex2 = "C:\\Dir";

        // Use ProcessStartInfo class.
        ProcessStartInfo startInfo = new ProcessStartInfo();
        startInfo.CreateNoWindow = false;
        startInfo.UseShellExecute = false;
        startInfo.FileName = "dcm2jpg.exe";
        startInfo.WindowStyle = ProcessWindowStyle.Hidden;
        startInfo.Arguments = "-f j -o \"" + ex1 + "\" -z 1.0 -s y " + ex2;

        try
        {
            // Start the process with the info we specified.
            // Call WaitForExit and then the using-statement will close.
            using (Process exeProcess = Process.Start(startInfo))
            {
                exeProcess.WaitForExit();
            }
        }
        catch
        {
            // Log error.
        }
    }
}

Extract Pages From a PDF

https://www.linuxjournal.com/content/tech-tip-extract-pages-pdf

There are a number of ways to extract a range of pages from a PDF file: there are PDF related toolkits for doing it, or you can use Ghostscript directly.
For example, to extract pages 22-36 from a 100-page PDF file using pdftk:
  $ pdftk A=100p-inputfile.pdf cat A22-36 output outfile_p22-p36.pdf
Or use a combination of xpdf-utils (or poppler-tools) with psutils and the ps2pdf command (which ships as part of Ghostscript):
  $ pdftops 100p-inputfile.pdf - | psselect -p22-36 | \
         ps2pdf14 - outfile_p22-p36.pdf
Or, just use Ghostscript (which, unlike pdftk, is installed nearly everywhere; and you've been using it in the last command anyway):
  $ gs -sDEVICE=pdfwrite -dNOPAUSE -dBATCH -dSAFER \
       -dFirstPage=22 -dLastPage=36 \
       -sOutputFile=outfile_p22-p36.pdf 100p-inputfile.pdf
Regarding speed and efficiency of the processing and more important the quality of the output file, the 2nd method above is for sure the worst of the 3. The conversion of the original PDF to PostScript and back to PDF (also known as "refrying" the PDF) is very unlikely to completely preserve advanced PDF features (such as transparency information, font hinting, overprinting information, color profiles, trapping instructions, etc.).
The 3rd method uses Ghostscript only (which the 2nd one uses anyway, because ps2pdf14 is nothing more than a wrapper script around a more or less complicated Ghostscript commandline. The 3rd method also preserves all the important PDF objects on your pages as they are, without any "roundtrip" conversions....
The only drawback of the 3rd method is that it's a longer, more complicated command line to type. But you can overcome that drawback if you save it as a bash function. Just put these lines in your ~/.bashrc file:
function pdfpextr()
{
    # this function uses 3 arguments:
    #     $1 is the first page of the range to extract
    #     $2 is the last page of the range to extract
    #     $3 is the input file
    #     output file will be named "inputfile_pXX-pYY.pdf"
    gs -sDEVICE=pdfwrite -dNOPAUSE -dBATCH -dSAFER \
       -dFirstPage=${1} \
       -dLastPage=${2} \
       -sOutputFile=${3%.pdf}_p${1}-p${2}.pdf \
       ${3}
}
Now you only need to type (after starting a new copy bash or sourcing .bashrc) the following:
  $ pdfpextr 22 36 inputfile.pdf
which will result in the file inputfile_p22-p36.pdf in the same directory as the input file.

Monday 23 April 2018

Online Scrum Tools – Part 3 – Import Backlog into Trello

Online Scrum Tools – Part 3 – Import Backlog into Trello

http://www.littlebluemonkey.com/blog/online-scrum-tools-part-3-upload-existing-product-backlog-into-trello

https://docs.google.com/spreadsheets/d/1gCBnYeK3f6uXBAV0jPYfFFFcZEjjeKoJ8Fg2h1PLG90/edit?pli=1#gid=680968806

In the previous post about my search for online Scrum tools, I extolled the virtues of Trello and how, for me, it's the perfect tool to manage a product backlog.
But what if you already have an existing backlog, and don't want to spend time re-keying the entries? Well luckily, like most things in 2012, there's an App for that.
Before you get too excited, I should mention that "The App" in this case is a Google Spreadsheet, with a Google Apps Script that leverages the Trello API to import the data. It's simple, effective, and in true agile fashion, it does just enough to get the job done without unnecessary bells and whistles.
So how does it work? Well make a copy of the spreadsheet in your Google Docs/Drive folder and get to work setting it up.

Go to the "Control" sheet and follow the instructions to set up API keys and tokens. Use the options in the Trello menu in the spreadsheet to find and record the IDs of the board and list that you want to add the cards too.
One of the steps is to add your entries into the "Backlog" sheet, the columns of which are worth looking at a little more closely:
  • Upload Status - Set to blank if you want the row to be imported. It is changed by the script as it processes each row.
  • Title - The description of the backlog item. This becomes the main text on the generated Trello card.
  • User Story - The user story (e.g. As a I want to so that ) for the backlog item. This gets recorded in the description field on the Trello card.
  • Points - The number of story points that have been assigned the story. The script puts this at the start of the title in parentheses. This is how the Trello Scrum Chrome extension handles story points.
  • Acceptance Criteria - The acceptance criteria for the story. These will be added as a single checklist on the Trello card, with a new line character separating the checklist items.
  • Comments - These will be added to the card as comments, with a new line character denoting a new comment.
  • Due Date - This will set the due date of the card. Date formats can be a problem, so it's best to use ISO Format if you can.
  • Labels - Enter a comma separated list of label that you want added to the card. 

Once the script runs, and if all goes well, you will end up with a list full of Trello cards that look something like this:

That's more or less it, easy right? There are, however, a few minor things that are worth noting:
  • Google Apps Scripts appear to time out at around 6 minutes, so I changed the import script to end gracefully after 5 1/2 minutes. The stories that have been imported will have "Completed" recorded in column A, so you can just repeatedly run the import script until all entries are imported.
  • I did notice that if you had the Trello board open during the import, then some checklists/comments did not immediately appear on the cards - refresh your browser page, and they should appear.
  • The Trello cards all have a card number on the bottom right, and it is this unique ID that we use if we don't want to refer to the story by name.
  • The performance seems to vary depending on the number of comments/acceptance criteria, but I was managing to import around 50-70 items in every import.

So there you have it - if you've skipped to the end of this post to find the good stuff (i.e. the link to the spreadsheet), then you're in luck - you'll find the spreadsheet here.

** Update : I've added a new post on how you can now go the other direction and  back up your Trello data in a Google Spreadsheet.
*** Update 7th November 2013 - If you do import a backlog into Trello, then there is another script available to Pimp Your Trello Card and automatically add Google Docs to your cards, Checklists, Descriptions, or even assign costs to your cards. ***
*** Update 7th January 2015 - I've made a few improvements to the script - details here. ***
*** Update 3rd November 2015 - another new version to address an api deprecated by Trello ***

MySQL 8.0: The end of MyISAM

MySQL 8.0: The end of MyISAM

  | October 11, 2016 |  Posted In: MySQL
MyISAMThis blog discusses the gradual end of MyISAM in MySQL.
The story that started 20 years ago is coming to its end. I’m talking about the old MyISAM storage engine that was the only storage provided by MySQL in 1995, and was available in MySQL for 20+ years. Actually, part of my job as a MySQL consultant for 10+ years was to discover MyISAM tables and advise customers how to convert those to InnoDB.
(Check your MySQL installation, you may still have MyISAM tables).
MySQL 5.7 still used MyISAM storage for the system tables in the MySQL schema.
In MySQL 8.0 (DMR version as of writing), the MyISAM storage engine is still available. But in a very limited scope:
  • After introducing the new data dictionary, the MyISAM tables are gone from the system schema (“mysql” db).
  • Working with MyISAM is harder now (and discouraged): you can’t just copy MyISAM tables into a running MySQL server, they will not be discovered (unlike InnoDB, where you can use “ALTER TABLE … IMPORT TABLESPACE”)
  • However, you can create a table engine=MyISAM, and it will work as before
InnoDB implemented all the older, missing features:
FeatureMyISAMInnoDB
Full Text IndexesyesSince MySQL 5.6
Portable tables (tablespaces)yesSince MySQL 5.6
Spatial Indexes/RTREE (GIS)yesSince MySQL 5.7
Last update for tableyesSince MySQL 5.7
(http://dev.mysql.com/worklog/task/?id=6658)
Suitable for temp tablesyesSince MySQL 5.7
Also complex selects uses InnoDBondisk temp tables
Faster count(*)yes*Faster in MySQL 5.7 but does not store counter

So the only MyISAM advantages left are:
  1. Tables will be smaller on disk compared to uncompressed InnoDB tables.
  2. The count(*) is still much faster in MyISAM:
I would not use MyISAM unless there is a specific case, and for well-known reasons (MyISAM are non-transactional, table level locks, with no crash recovery, etc.)
My colleague Laurynas Biveinis also suggested converting MyISAM to an optional storage engine plugin.