-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathdocs-docker.html
789 lines (473 loc) · 47.3 KB
/
docs-docker.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
<!DOCTYPE html>
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content=" Singularity and Docker Previous Next Import a Docker image into a Singularity ImageThe core of a Docker image is basically a compressed set of files, a set of .tar.gz that (if you look in your Docker image folder on your host machine, you will see. We are going to use this local repository for this first set of methods.Quick Start: Use the Docker Remote APIImport Docker to SingularityThe Docker engine communicates with the Docker Hub via the Docker Remote API, and guess what, we can too! The easiest thing to do is create an image, and then pipe a Docker image directly into it from the Docker Registry. This first method does not require having Docker installed on your machine. Let’s say that I want to bootstrap tensorflow from Docker. First I should create the tensorflow image:sudo singularity create --size 4000 tensorflow.imgsudo singularity import tensorflow.img docker://tensorflow/tensorflow:latestCache folder set to /home/vanessa/.singularity/dockerExtracting /home/vanessa/.singularity/docker/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4.tar.gzExtracting /home/vanessa/.singularity/docker/sha256:182b64c1f020de1cb4b2783b3a13fbeb07ec4087bc911352d0f5ef40c8eec8cf.tar.gzExtracting /home/vanessa/.singularity/docker/sha256:bfc1d5e3de1cf70353afb2b81fbbeab16bad961352b86f60901bc1da1396f2b4.tar.gzExtracting /home/vanessa/.singularity/docker/sha256:d819f1ec59a06c001b37e66dd1639c591e606029ea7584fac704ff741cda249b.tar.gzExtracting /home/vanessa/.singularity/docker/sha256:f5d83de9c6786bff4160679ed4bde332970367225ede609944bbe686edb1c25b.tar.gzExtracting /home/vanessa/.singularity/docker/sha256:c1f8f4c880d49d70a8280860e3bc5ee559a95d4e1dc44f9128b638eb2240324c.tar.gzExtracting /home/vanessa/.singularity/docker/sha256:9528c5352798ec3a134be13b66bc4dc71e7cdd029e268ae3cdfeb0719a4c8b8b.tar.gzExtracting /home/vanessa/.singularity/docker/sha256:c145c1f339f57690f80bd64e86caa3b00e0635a6a383bc8be7726a3baf22a0d2.tar.gzExtracting /home/vanessa/.singularity/docker/sha256:ebb77ce6e1c6769c1849194c1319dc6978e19575c76fd1fa942a623b6f2996a4.tar.gzExtracting /home/vanessa/.singularity/docker/sha256:51900bc9e720db035e12f6c425dd9c06928a9d1eb565c86572b3aab93d24cfca.tar.gzExtracting /home/vanessa/.singularity/docker/sha256:f8419ea7c1b5d667cf26c2c5ec0bfb3502872e5afc6aa85caf2b8c7650bdc8d9.tar.gzExtracting /home/vanessa/.singularity/docker/sha256:3eed5ff20a90a40b0cb7909e79128740f1320d29bec2ae9e025a1d375555db15.tar.gzExtracting /home/vanessa/.singularity/docker/sha256:6c953ac5d795ea26fd59dc5bdf4d335625c69f8bcfbdd8307d6009c2e61779c9.tar.gzAdding Docker CMD as Singularity runscript.../run_jupyter.shBootstrap initializationNo bootstrap definition passed, updating containerExecuting Prebootstrap moduleExecuting Postbootstrap moduleDone.Note that if you want (much) more detailed output for debugging to the console, you need to enable --verbose mode:sudo singularity --verbose import tensorflow.img docker://tensorflow/tensorflow:latestNow I can shell into it, and import tensorflow:$ singularity shell tensorflow.img Singularity: Invoking an interactive shell within container...Singularity.tensorflow.img> lsSingularity.tensorflow.img> pythonPython 2.7.6 (default, Jun 22 2015, 17:58:13) [GCC 4.8.2] on linux2Type "help", "copyright", "credits" or "license" for more information.>>> import tensorflow>>> How does the runscript work?Docker has two commands in the DOCKERFILE that have something to do with execution, CMD and ENTRYPOINT. The differences are subtle, but the best description I’ve found is the following: A CMD is to provide defaults for an executing container. and An ENTRYPOINT helps you to configure a container that you can run as an executable. Given the definition, the ENTRYPOINT is most appropriate for the Singularity %runscript, and so using the default bootstrap (whether from a docker:// endpoint or a Singularity spec file) will set the ENTRYPOINT variable as the runscript. You can change this behavior by specifying IncludeCmd: yes in the Spec file (see below). If you provide any sort of %runscript in your Spec file, this overrides anything provided in Docker. In summary, the order of operations is as follows: If a %runscript is specified in the Singularity spec file, this takes prevalence over all If no %runscript is specified, or if the import command is used as in the example above, the ENTRYPOINT is used as runscript. If no %runscript is specified, but the user has a Singularity spec with IncludeCmd, then the Docker CMD is used.Use a Spec FileDo a barrel role! Use a spec file! Many times, you want to bootstrap an image, and then either change the %runscript or add additional software or commands in the %post section. To achieve this, you can create a specification file. Currently, these are distributed with the naming format [myfile].def, however (soon) we will use a standard name, Singularity so all specification files can be automatically found. Here is what the spec file would look like for tensorflow:Bootstrap: dockerFrom: tensorflow/tensorflow:latest%runscript exec /usr/bin/python "$@"%post echo "Post install stuffs!"In the example above, I am overriding any Dockerfile ENTRYPOINT because I have defined a %runscript. If I want the Dockerfile ENTRYPOINT to take preference, I would remove the %runscript section:Bootstrap: dockerFrom: tensorflow/tensorflow:latest%post echo "Post install stuffs!"Note that the spec file above would be (almost) equivalent to the command:sudo singularity import tensorflow.img docker://tensorflow/tensorflow:latestminus the useless echo at the end. If I want the CMD to take preference, I would add IncludeCmd:Bootstrap: dockerFrom: tensorflow/tensorflow:latestIncludeCmd: yes%post echo "Post install stuffs!"The solutions above would be ideal for saving a custom specification of an image to build at some runtime.Custom AuthenticationFor both import and bootstrap using a build spec file, by default we use the Docker Registry index.docker.io. Singularity first tries the call without a token, and then asks for one with pull permissions if the request is defined. However, it may be the case that you want to provide a custom token for a private registry. You have two options. You can either provide a Username and Password in the build specification file (if stored locally and there is no need to share), or (in the case of doing an import or needing to secure the credentials) you can export these variables to environmental variables. We provide instructions for each of these cases:Authentication in the Spec FileYou can simply specify your additional authentication parameters in the header with the labels Username and Password:Username: vanessaPassword: [password]Again, this can be in addition to specification of a custom registry with the Registry parameter.Authentication in the EnvironmentYou can export your registry, username, and password for Singularity as follows:export SINGULARITY_DOCKER_REGISTRY='--registry myrepo'export SINGULARITY_DOCKER_AUTH='--username vanessa --password [password]'Testing AuthenticationIf you are having trouble, you can test your token by obtaining it on the command line and putting it into an environmental variable, CREDENTIAL:CREDENTIAL=$(echo -n vanessa:[password] | base64)TOKEN=$(http 'https://auth.docker.io/token?service=registry.docker.io&scope=repository:vanessa/code-samples:pull' Authorization:"Basic $CREDENTIAL" | jq -r '.token')This should place the token in the environmental variable TOKEN. To test that your token is valid, you can do the followinghttp https://index.docker.io/v2/vanessa/code-samples/tags/list Authorization:"Bearer $TOKEN"The above call should return the tags list as expected.Run a Singularity Shell from a Docker imageFinally, we can achieve a “shell” experience, meaning shelling into Docker image imported into Singularity. We do this by storing the entire image in a temporary location, and then running the same function. You would do something like this:sudo singularity shell docker://ubuntu:latestDetailed Start: Bootstrapping a Docker imageA common use case is to want to start with a Docker image, possibly add custom commands, and have a Singularity image when you finish. You can read a bit about bootstrapping here to get a sense of adding the custom commands and software. To specify “Docker” as the build source, you simply need this header:Bootstrap: dockerFrom: ubuntu:latestIncludeCmd: yes Boostrap: docker specifies that you want to import from Docker. This is required for Docker bootstrapping! IncludeCmd: will add the Dockerfile CMD as the runscript (/singularity) if one is found. If you define a different one later in the spec file, your specification will overwrite this one. From: works the same way as it does for Docker. You can specify, maximally, a library/imagename:tag. For example, all of the following are valid: library/ubuntu:latest library/ubuntu ubuntu:latest ubuntu In the case of omitting the tag (latest) it is assumed that you want the latest image. In the case of omitting the namespace (library) it is assumed that you want the common namespace, library. If you have a reason to use the Docker Engine, we also have a method to do this. The benefit of this method would be that you could use an image built locally (in your local cache) that isn’t on Docker Hub.Using Docker EngineHere we will access Docker images via the Docker command line tool, meaning using the Docker engine. As is the Docker standard, the image is first looked for in your local cache, and if not found, is pulled from Docker Hub.docker2singularity.sh: DockerizedWe wrapped this entire process into a Docker container itself, which means that you can use a Docker container in a Docker container to export a Docker container into Singularity! Nuts. Full instructions are provided, however here is the gist: docker run \ -v /var/run/docker.sock:/var/run/docker.sock \ -v D:\host\path\where\to\ouptut\singularity\image:/output \ --privileged -t --rm \ singularityware/docker2singularity \ ubuntu:14.04How does docker2singularity.sh work?How did this come to be? It so happens that Docker has an “export” command to pipe this data out, and Singularity has an “import” command to take them in. Thus, you can do a simple import of a Docker image into a Singularity command by doing:# Here is the name of the Singularity image I will createimage=ubuntu.img# Now I am creating itsudo singularity create $image# Now I am exporting a running Docker container into it via a pipe (|)docker export $container_id | singularity import $imageWhere $container_id is the id of a running container obtained with docker ps. However, there are subtle details like the environment and permissions that this method will miss. It’s also the case that most Docker images don’t run (and stay running) easily unless you do something like:docker run -d $image tail -f /dev/nullEarly on we created a docker2singularity.sh, a script that you can download and run as follows:wget https://raw.githubusercontent.com/singularityware/docker2singularity/master/docker2singularity.shchmod u+x docker2singularity.sh./docker2singularity.sh ubuntu:latestTo produce a Singularity image of “ubuntu:latest” in the present working directory.TroubleshootingWhy won’t my image bootstrap work? If you can’t find an answer on this site, please ping us an issue.If you’ve found an answer and you’d like to see it on the site for others to benefit from, then post to us here.FutureThis entire process will hopefully change in two ways. First, we hope to collapse the image creation and bootstrapping, so you have the option to do them both in one full swing. Second, we hope to eventually figure out some kind of solution to import Docker containers without needing sudo. Previous Next Edit me Site last generated: Jul 26, 2017 ">
<meta name="name" content="Singularity and Docker">
<meta name="thumbnail" content="http://singularity.lbl.gov/images/logo/logo.svg">
<title>Singularity and Docker | Singularity</title>
<link rel="stylesheet" href="assets/css/syntax.css">
<link rel="stylesheet" type="text/css" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.5.0/css/font-awesome.min.css">
<!--<link rel="stylesheet" type="text/css" href="css/bootstrap.min.css">-->
<link rel="stylesheet" href="assets/css/modern-business.css">
<link rel="stylesheet" href="assets/css/lavish-bootstrap.css">
<link rel="stylesheet" href="assets/css/customstyles.css">
<link rel="stylesheet" href="assets/css/theme-blue.css">
<link rel="stylesheet" type="text/css" href="assets/css/asciinema-player.css" />
<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/2.1.4/jquery.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery-cookie/1.4.1/jquery.cookie.min.js"></script>
<script src="assets/js/jquery.navgoco.min.js"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/js/bootstrap.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/anchor-js/2.0.0/anchor.min.js"></script>
<script src="assets/js/toc.js"></script>
<script src="assets/js/customscripts.js"></script>
<link rel="shortcut icon" href="images/favicon/favicon.ico">
<!-- HTML5 Shim and Respond.js IE8 support of HTML5 elements and media queries -->
<!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
<!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
<script src="https://oss.maxcdn.com/libs/respond.js/1.4.2/respond.min.js"></script>
<![endif]-->
<link rel="alternate" type="application/rss+xml" title="" href="http://localhost:4005feed.xml">
<script>
$(document).ready(function() {
// Initialize navgoco with default options
$("#mysidebar").navgoco({
caretHtml: '',
accordion: true,
openClass: 'active', // open
save: false, // leave false or nav highlighting doesn't work right
cookie: {
name: 'navgoco',
expires: false,
path: '/'
},
slide: {
duration: 400,
easing: 'swing'
}
});
$("#collapseAll").click(function(e) {
e.preventDefault();
$("#mysidebar").navgoco('toggle', false);
});
$("#expandAll").click(function(e) {
e.preventDefault();
$("#mysidebar").navgoco('toggle', true);
});
});
</script>
<script>
$(function () {
$('[data-toggle="tooltip"]').tooltip()
})
</script>
</head>
<body>
<!-- asciinema player -->
<script src="assets/js/asciinema-player.js"></script>
<!-- Show or hide players on button clicks-->
<script>
$( document ).ready(function() {
$(".asciinema-button").click(function(){
var asciinemaVideo = "#asciinema-" + $(this).attr('id');
if ($(asciinemaVideo).hasClass('hidden')){
$(asciinemaVideo).removeClass('hidden');
$(this).text('Hide Tutorial')
} else {
$(asciinemaVideo).addClass('hidden');
$(this).text('Show Tutorial')
}
});
});
</script>
<!-- Navigation -->
<nav class="navbar navbar-inverse navbar-fixed-top">
<div class="container topnavlinks">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1">
<span class="sr-only">Toggle navigation</span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="index.html"> <span class="projectTitle"> Singularity</span></a>
</div>
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav navbar-right">
<!-- entries without drop-downs appear here -->
<li><a href="blog">News</a></li>
<!-- entries with drop-downs appear here -->
<!-- conditional logic to control which topnav appears for the audience defined in the configuration file.-->
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Docs<b class="caret"></b></a>
<ul class="dropdown-menu">
<li><a href="admin-guide">Admin Guide</a></li>
<li><a href="user-guide">User Guide</a></li>
<li><a href="links">Contributed Content</a></li>
</ul>
</li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Quick Links<b class="caret"></b></a>
<ul class="dropdown-menu">
<li><a href="https://github.com/singularityware/singularity" target="_blank">Github Repo</a></li>
<li><a href="https://groups.google.com/a/lbl.gov/forum/#!forum/singularity" target="_blank">Google Group</a></li>
<li><a href="http://stackoverflow.com/questions/tagged/singularity" target="_blank">Singularity on Stack Overflow</a></li>
<li><a href="https://singularity-hub.org/faq" target="_blank">Singularity Hub</a></li>
<li><a href="https://singularity-container.slack.com" target="_blank">Slack</a></li>
<li><a href="faq#troubleshooting">Troubleshooting</a></li>
</ul>
</li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">People<b class="caret"></b></a>
<ul class="dropdown-menu">
<li><a href="http://www.lbl.gov/" target="_blank">LBNL</a></li>
<li><a href="https://srcc.stanford.edu/" target="_blank">Stanford</a></li>
</ul>
</li>
<li><a href="/search"><i class="fa fa-search"></i></li>
<!-- jekyll search hidden in favor of google
<li>
<div id="search-demo-container">
<input type="text" id="search-input" placeholder="search...">
<ul id="results-container"></ul>
</div>
<script src="assets/js/jekyll-search.js" type="text/javascript"></script>
<script type="text/javascript">
SimpleJekyllSearch.init({
searchInput: document.getElementById('search-input'),
resultsContainer: document.getElementById('results-container'),
dataSource: 'search.json',
searchResultTemplate: '<li><a href="{url}" title="Singularity and Docker">{title}</a></li>',
noResultsText: 'No results found.',
limit: 10,
fuzzy: true,
})
</script>
end search-->
</li>
</ul>
</div>
</div>
<!-- /.container -->
</nav>
<!-- Page Content -->
<div class="container">
<div class="col-lg-12"> </div>
<!-- Content Row -->
<div class="row">
<!-- Sidebar Column -->
<div class="col-md-3">
<div class="shiny"><a href="\"><figure><img src="/images/logo/logo.svg" class="sidebar-logo"/></figure></a></div>
<ul id="mysidebar" class="nav">
<li class="sidebarTitle">User Guide</li>
<li>
<a href="#">Getting Started</a>
<ul>
<li><a href="user-guide">Introduction</a></li>
<li><a href="docs-quick-start-installation">Quick Start Installation</a></li>
<li><a href="bootstrap-image">Bootstrap an Image</a></li>
<li><a href="create-image">Create an Image</a></li>
<li><a href="docs-mount">Bind Paths and Files</a></li>
<li><a href="docs-changing-containers">Change an Existing Container</a></li>
<li class="active"><a href="docs-docker">Singularity and Docker</a></li>
<li><a href="faq#troubleshooting">Troubleshooting</a></li>
</ul>
<li>
<a href="#">Commands</a>
<ul>
<li><a href="docs-usage">Command Usage</a></li>
<li><a href="docs-bootstrap">bootstrap</a></li>
<li><a href="docs-exec">exec</a></li>
<li><a href="docs-export">export</a></li>
<li><a href="docs-import">import</a></li>
<li><a href="docs-shell">shell</a></li>
<li><a href="docs-run">run</a></li>
</ul>
<!-- if you aren't using the accordion, uncomment this block:
<p class="external">
<a href="#" id="collapseAll">Collapse All</a> | <a href="#" id="expandAll">Expand All</a>
</p>
-->
</li>
</ul>
</div>
<!-- this highlights the active parent class in the navgoco sidebar. this is critical so that the parent expands when you're viewing a page. This must appear below the sidebar code above. Otherwise, if placed inside customscripts.js, the script runs before the sidebar code runs and the class never gets inserted.-->
<script>$("li.active").parents('li').toggleClass("active");</script>
<!-- Content Column -->
<div class="col-md-9">
<div class="post-header">
<h1 class="post-title-main">Singularity and Docker</h1>
</div>
<div class="post-content">
<!-- Previous and next buttons-->
<div class="row" style="padding-top:30px; margin-bottom:10px"><div class="col-md-12">
<a href="#"><button style="width:20%; height: 70px; float:left" class="hidden previous-button btn btn-lg btn-default">Previous</button></a>
<a href="#"><button style="width:20%; height: 70px; float:right" class="hidden next-button btn btn-lg btn-default">Next</button></a>
</div></div>
<script>
$(document).ready(function(){
var next = $("li.active").next().last().find('a').attr('href');
var previous = $("li.active").prev().last().find('a').attr('href');
// NEXT BUTTON
if (typeof next == 'undefined'){
console.log("disabling next button")
$(".next-button").addClass("hidden")
} else if (next == "#") {
next = $("li.active").next().find("li").first().find('a').attr('href');
$(".next-button").closest('a').attr('href', next)
$(".next-button").removeClass('hidden')
} else {
$(".next-button").closest('a').attr('href', next)
$(".next-button").removeClass('hidden')
}
// PREVIOUS BUTTON
if (typeof previous == 'undefined'){
console.log("disabling previous button")
$(".previous-button").addClass("hidden")
} else if (previous == "#") {
previous = $("li.active").prev().find("li").last().find('a').attr('href')
$(".previous-button").closest('a').attr('href', previous)
$(".previous-button").removeClass('hidden')
} else {
$(".previous-button").closest('a').attr('href', previous)
$(".previous-button").removeClass('hidden')
}
})
</script>
<!-- this handles the automatic toc. use ## for subheads to auto-generate the on-page minitoc. if you use html tags, you must supply an ID for the heading element in order for it to appear in the minitoc. -->
<script>
$( document ).ready(function() {
// Handler for .ready() called.
$('#toc').toc({ minimumHeaders: 0, listType: 'ul', showSpeed: 0, headers: 'h2,h3,h4' });
/* this offset helps account for the space taken up by the floating toolbar. */
$('#toc').on('click', 'a', function() {
var target = $(this.getAttribute('href'))
, scroll_target = target.offset().top
$(window).scrollTop(scroll_target - 10);
return false
})
});
</script>
<div id="toc"></div>
<h1 id="import-a-docker-image-into-a-singularity-image">Import a Docker image into a Singularity Image</h1>
<p>The core of a Docker image is basically a compressed set of files, a set of <code class="highlighter-rouge">.tar.gz</code> that (if you look in your <a href="http://stackoverflow.com/questions/19234831/where-are-docker-images-stored-on-the-host-machine" target="_blank">Docker image folder</a> on your host machine, you will see. We are going to use this local repository for this first set of methods.</p>
<h2 id="quick-start-use-the-docker-remote-api">Quick Start: Use the Docker Remote API</h2>
<h3 id="import-docker-to-singularity">Import Docker to Singularity</h3>
<p>The Docker engine communicates with the Docker Hub via the <a href="https://docs.docker.com/engine/reference/api/docker_remote_api/" target="_blank">Docker Remote API</a>, and guess what, we can too! The easiest thing to do is create an image, and then pipe a Docker image directly into it from the Docker Registry. This first method does not require having Docker installed on your machine. Let’s say that I want to bootstrap tensorflow from Docker. First I should create the tensorflow image:</p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code>sudo singularity create --size 4000 tensorflow.img
sudo singularity import tensorflow.img docker://tensorflow/tensorflow:latest
Cache folder <span class="nb">set </span>to /home/vanessa/.singularity/docker
Extracting /home/vanessa/.singularity/docker/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4.tar.gz
Extracting /home/vanessa/.singularity/docker/sha256:182b64c1f020de1cb4b2783b3a13fbeb07ec4087bc911352d0f5ef40c8eec8cf.tar.gz
Extracting /home/vanessa/.singularity/docker/sha256:bfc1d5e3de1cf70353afb2b81fbbeab16bad961352b86f60901bc1da1396f2b4.tar.gz
Extracting /home/vanessa/.singularity/docker/sha256:d819f1ec59a06c001b37e66dd1639c591e606029ea7584fac704ff741cda249b.tar.gz
Extracting /home/vanessa/.singularity/docker/sha256:f5d83de9c6786bff4160679ed4bde332970367225ede609944bbe686edb1c25b.tar.gz
Extracting /home/vanessa/.singularity/docker/sha256:c1f8f4c880d49d70a8280860e3bc5ee559a95d4e1dc44f9128b638eb2240324c.tar.gz
Extracting /home/vanessa/.singularity/docker/sha256:9528c5352798ec3a134be13b66bc4dc71e7cdd029e268ae3cdfeb0719a4c8b8b.tar.gz
Extracting /home/vanessa/.singularity/docker/sha256:c145c1f339f57690f80bd64e86caa3b00e0635a6a383bc8be7726a3baf22a0d2.tar.gz
Extracting /home/vanessa/.singularity/docker/sha256:ebb77ce6e1c6769c1849194c1319dc6978e19575c76fd1fa942a623b6f2996a4.tar.gz
Extracting /home/vanessa/.singularity/docker/sha256:51900bc9e720db035e12f6c425dd9c06928a9d1eb565c86572b3aab93d24cfca.tar.gz
Extracting /home/vanessa/.singularity/docker/sha256:f8419ea7c1b5d667cf26c2c5ec0bfb3502872e5afc6aa85caf2b8c7650bdc8d9.tar.gz
Extracting /home/vanessa/.singularity/docker/sha256:3eed5ff20a90a40b0cb7909e79128740f1320d29bec2ae9e025a1d375555db15.tar.gz
Extracting /home/vanessa/.singularity/docker/sha256:6c953ac5d795ea26fd59dc5bdf4d335625c69f8bcfbdd8307d6009c2e61779c9.tar.gz
Adding Docker CMD as Singularity runscript...
/run_jupyter.sh
Bootstrap initialization
No bootstrap definition passed, updating container
Executing Prebootstrap module
Executing Postbootstrap module
Done.
</code></pre>
</div>
<p>Note that if you want (much) more detailed output for debugging to the console, you need to enable <code class="highlighter-rouge">--verbose</code> mode:</p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code>sudo singularity --verbose import tensorflow.img docker://tensorflow/tensorflow:latest
</code></pre>
</div>
<p>Now I can shell into it, and import tensorflow:</p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code><span class="gp">$ </span>singularity shell tensorflow.img
Singularity: Invoking an interactive shell within container...
<span class="gp">Singularity.tensorflow.img> </span>ls
<span class="gp">Singularity.tensorflow.img> </span>python
Python 2.7.6 <span class="o">(</span>default, Jun 22 2015, 17:58:13<span class="o">)</span>
<span class="o">[</span>GCC 4.8.2] on linux2
Type <span class="s2">"help"</span>, <span class="s2">"copyright"</span>, <span class="s2">"credits"</span> or <span class="s2">"license"</span> <span class="k">for </span>more information.
<span class="gp">>>> </span>import tensorflow
<span class="gp">>>> </span>
</code></pre>
</div>
<h3 id="how-does-the-runscript-work">How does the runscript work?</h3>
<p>Docker has two commands in the <code class="highlighter-rouge">DOCKERFILE</code> that have something to do with execution, <code class="highlighter-rouge">CMD</code> and <code class="highlighter-rouge">ENTRYPOINT</code>. The differences are subtle, but the best description I’ve found is the following:</p>
<blockquote>
<blockquote>
<p>A <code class="highlighter-rouge">CMD</code> is to provide defaults for an executing container.</p>
</blockquote>
</blockquote>
<p>and</p>
<blockquote>
<blockquote>
<p>An <code class="highlighter-rouge">ENTRYPOINT</code> helps you to configure a container that you can run as an executable.</p>
</blockquote>
</blockquote>
<p>Given the definition, the <code class="highlighter-rouge">ENTRYPOINT</code> is most appropriate for the Singularity <code class="highlighter-rouge">%runscript</code>, and so using the default bootstrap (whether from a <code class="highlighter-rouge">docker://</code> endpoint or a <code class="highlighter-rouge">Singularity</code> spec file) will set the <code class="highlighter-rouge">ENTRYPOINT</code> variable as the runscript. You can change this behavior by specifying <code class="highlighter-rouge">IncludeCmd: yes</code> in the Spec file (see below). If you provide any sort of <code class="highlighter-rouge">%runscript</code> in your Spec file, this overrides anything provided in Docker. In summary, the order of operations is as follows:</p>
<ol>
<li>If a <code class="highlighter-rouge">%runscript</code> is specified in the <code class="highlighter-rouge">Singularity</code> spec file, this takes prevalence over all</li>
<li>If no <code class="highlighter-rouge">%runscript</code> is specified, or if the <code class="highlighter-rouge">import</code> command is used as in the example above, the <code class="highlighter-rouge">ENTRYPOINT</code> is used as runscript.</li>
<li>If no <code class="highlighter-rouge">%runscript</code> is specified, but the user has a <code class="highlighter-rouge">Singularity</code> spec with <code class="highlighter-rouge">IncludeCmd</code>, then the Docker <code class="highlighter-rouge">CMD</code> is used.</li>
</ol>
<h3 id="use-a-spec-file">Use a Spec File</h3>
<p>Do a barrel role! Use a spec file! Many times, you want to bootstrap an image, and then either change the <code class="highlighter-rouge">%runscript</code> or add additional software or commands in the <code class="highlighter-rouge">%post</code> section. To achieve this, you can create a specification file. Currently, these are distributed with the naming format <code class="highlighter-rouge">[myfile].def</code>, however (soon) we will use a standard name, <code class="highlighter-rouge">Singularity</code> so all specification files can be automatically found. Here is what the spec file would look like for tensorflow:</p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code>Bootstrap: docker
From: tensorflow/tensorflow:latest
%runscript
<span class="nb">exec</span> /usr/bin/python <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
%post
<span class="nb">echo</span> <span class="s2">"Post install stuffs!"</span>
</code></pre>
</div>
<p>In the example above, I am overriding any Dockerfile <code class="highlighter-rouge">ENTRYPOINT</code> because I have defined a <code class="highlighter-rouge">%runscript</code>. If I want the Dockerfile <code class="highlighter-rouge">ENTRYPOINT</code> to take preference, I would remove the <code class="highlighter-rouge">%runscript</code> section:</p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code>Bootstrap: docker
From: tensorflow/tensorflow:latest
%post
<span class="nb">echo</span> <span class="s2">"Post install stuffs!"</span>
</code></pre>
</div>
<p>Note that the spec file above would be (almost) equivalent to the command:</p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code>sudo singularity import tensorflow.img docker://tensorflow/tensorflow:latest
</code></pre>
</div>
<p>minus the useless echo at the end. If I want the <code class="highlighter-rouge">CMD</code> to take preference, I would add <code class="highlighter-rouge">IncludeCmd</code>:</p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code>Bootstrap: docker
From: tensorflow/tensorflow:latest
IncludeCmd: yes
%post
<span class="nb">echo</span> <span class="s2">"Post install stuffs!"</span>
</code></pre>
</div>
<p>The solutions above would be ideal for saving a custom specification of an image to build at some runtime.</p>
<h3 id="custom-authentication">Custom Authentication</h3>
<p>For both import and bootstrap using a build spec file, by default we use the Docker Registry <code class="highlighter-rouge">index.docker.io</code>. Singularity first tries the call without a token, and then asks for one with pull permissions if the request is defined. However, it may be the case that you want to provide a custom token for a private registry. You have two options. You can either provide a <code class="highlighter-rouge">Username</code> and <code class="highlighter-rouge">Password</code> in the build specification file (if stored locally and there is no need to share), or (in the case of doing an import or needing to secure the credentials) you can export these variables to environmental variables. We provide instructions for each of these cases:</p>
<h4 id="authentication-in-the-spec-file">Authentication in the Spec File</h4>
<p>You can simply specify your additional authentication parameters in the header with the labels <code class="highlighter-rouge">Username</code> and <code class="highlighter-rouge">Password</code>:</p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code>Username: vanessa
Password: <span class="o">[</span>password]
</code></pre>
</div>
<p>Again, this can be in addition to specification of a custom registry with the <code class="highlighter-rouge">Registry</code> parameter.</p>
<h4 id="authentication-in-the-environment">Authentication in the Environment</h4>
<p>You can export your registry, username, and password for Singularity as follows:</p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code><span class="nb">export </span><span class="nv">SINGULARITY_DOCKER_REGISTRY</span><span class="o">=</span><span class="s1">'--registry myrepo'</span>
<span class="nb">export </span><span class="nv">SINGULARITY_DOCKER_AUTH</span><span class="o">=</span><span class="s1">'--username vanessa --password [password]'</span>
</code></pre>
</div>
<h5 id="testing-authentication">Testing Authentication</h5>
<p>If you are having trouble, you can test your token by obtaining it on the command line and putting it into an environmental variable, <code class="highlighter-rouge">CREDENTIAL</code>:</p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code><span class="nv">CREDENTIAL</span><span class="o">=</span><span class="k">$(</span><span class="nb">echo</span> -n vanessa:[password] | base64<span class="k">)</span>
<span class="nv">TOKEN</span><span class="o">=</span><span class="k">$(</span>http <span class="s1">'https://auth.docker.io/token?service=registry.docker.io&scope=repository:vanessa/code-samples:pull'</span> Authorization:<span class="s2">"Basic </span><span class="nv">$CREDENTIAL</span><span class="s2">"</span> | jq -r <span class="s1">'.token'</span><span class="k">)</span>
</code></pre>
</div>
<p>This should place the token in the environmental variable <code class="highlighter-rouge">TOKEN</code>. To test that your token is valid, you can do the following</p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code>http https://index.docker.io/v2/vanessa/code-samples/tags/list Authorization:<span class="s2">"Bearer </span><span class="nv">$TOKEN</span><span class="s2">"</span>
</code></pre>
</div>
<p>The above call should return the tags list as expected.</p>
<h3 id="run-a-singularity-shell-from-a-docker-image">Run a Singularity Shell from a Docker image</h3>
<p>Finally, we can achieve a “shell” experience, meaning shelling into Docker image imported into Singularity. We do this by storing the entire image in a temporary location, and then running the same function. You would do something like this:</p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code>sudo singularity shell docker://ubuntu:latest
</code></pre>
</div>
<h2 id="detailed-start-bootstrapping-a-docker-image">Detailed Start: Bootstrapping a Docker image</h2>
<p>A common use case is to want to start with a Docker image, possibly add custom commands, and have a Singularity image when you finish. You can read a bit about <a href="/bootstrap-image" target="_blank">bootstrapping here</a> to get a sense of adding the custom commands and software. To specify “Docker” as the build source, you simply need this header:</p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code>Bootstrap: docker
From: ubuntu:latest
IncludeCmd: yes
</code></pre>
</div>
<ul>
<li>Boostrap: docker specifies that you want to import from Docker. This is required for Docker bootstrapping!</li>
<li>IncludeCmd: will add the Dockerfile CMD as the runscript (<code class="highlighter-rouge">/singularity</code>) if one is found. If you define a different one later in the spec file, your specification will overwrite this one.</li>
<li>From: works the same way <a href="https://docs.docker.com/engine/reference/builder/" target="_blank">as it does</a> for Docker. You can specify, maximally, a library/imagename:tag. For example, all of the following are valid:
<ul>
<li>library/ubuntu:latest</li>
<li>library/ubuntu</li>
<li>ubuntu:latest</li>
<li>ubuntu</li>
</ul>
</li>
</ul>
<p>In the case of omitting the tag (latest) it is assumed that you want the latest image. In the case of omitting the namespace (library) it is assumed that you want the common namespace, library. If you have a reason to use the Docker Engine, we also have a method to do this. The benefit of this method would be that you could use an image built locally (in your local cache) that isn’t on Docker Hub.</p>
<h3 id="using-docker-engine">Using Docker Engine</h3>
<p>Here we will access Docker images via the <code class="highlighter-rouge">Docker</code> command line tool, meaning using the Docker engine. As is the Docker standard, the image is first looked for in your local cache, and if not found, is pulled from Docker Hub.</p>
<h4 id="docker2singularitysh-dockerized">docker2singularity.sh: Dockerized</h4>
<p>We wrapped this entire process into a Docker container itself, which means that you can use a Docker container in a Docker container to export a Docker container into Singularity! Nuts. Full instructions <a href="https://github.com/singularityware/docker2singularity" target="_blank"> are provided, however here is the gist:</a></p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code> docker run <span class="se">\ </span>
-v /var/run/docker.sock:/var/run/docker.sock <span class="se">\</span>
-v D:<span class="se">\h</span>ost<span class="se">\p</span>ath<span class="se">\w</span>here<span class="se">\t</span>o<span class="se">\o</span>uptut<span class="se">\s</span>ingularity<span class="se">\i</span>mage:/output <span class="se">\</span>
--privileged -t --rm <span class="se">\</span>
singularityware/docker2singularity <span class="se">\ </span>
ubuntu:14.04
</code></pre>
</div>
<h5 id="how-does-docker2singularitysh-work">How does docker2singularity.sh work?</h5>
<p>How did this come to be? It so happens that Docker has an “export” command to pipe this data out, and Singularity has an “import” command to take them in. Thus, you can do a simple import of a Docker image into a Singularity command by doing:</p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code><span class="c"># Here is the name of the Singularity image I will create</span>
<span class="nv">image</span><span class="o">=</span>ubuntu.img
<span class="c"># Now I am creating it</span>
sudo singularity create <span class="nv">$image</span>
<span class="c"># Now I am exporting a running Docker container into it via a pipe (|)</span>
docker <span class="nb">export</span> <span class="nv">$container_id</span> | singularity import <span class="nv">$image</span>
</code></pre>
</div>
<p>Where <code class="highlighter-rouge">$container_id</code> is the id of a running container obtained with <code class="highlighter-rouge">docker ps</code>. However, there are subtle details like the environment and permissions that this method will miss. It’s also the case that most Docker images don’t run (and stay running) easily unless you do something like:</p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code>docker run -d <span class="nv">$image</span> tail -f /dev/null
</code></pre>
</div>
<p>Early on we created a <a href="https://github.com/singularityware/docker2singularity/blob/master/docker2singularity.sh" target="_blank">docker2singularity.sh</a>, a script that you can download and run as follows:</p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code>wget https://raw.githubusercontent.com/singularityware/docker2singularity/master/docker2singularity.sh
chmod u+x docker2singularity.sh
./docker2singularity.sh ubuntu:latest
</code></pre>
</div>
<p>To produce a Singularity image of “ubuntu:latest” in the present working directory.</p>
<h2 id="troubleshooting">Troubleshooting</h2>
<p>Why won’t my image bootstrap work? If you can’t find an answer on this site, please <a href="https://www.github.com/singularityware/singularity/issues" target="_blank">ping us an issue</a>.
If you’ve found an answer and you’d like to see it on the site for others to benefit from, then post to us <a href="https://www.github.com/singularityware/singularityware.github.io/issues" target="_blank">here</a>.</p>
<h2 id="future">Future</h2>
<p>This entire process will hopefully change in two ways. First, we hope to collapse the image creation and bootstrapping, so you have the option to do them both in one full swing. Second, we hope to eventually figure out some kind of solution to import Docker containers without needing sudo.</p>
<!-- More navigation on the bottom -->
<div class="row" style="padding-top:30px; margin-bottom:10px"><div class="col-md-12">
<a href="#"><button style="width:20%; height: 70px; float:left" class="hidden previous-button btn btn-lg btn-default">Previous</button></a>
<a href="#"><button style="width:20%; height: 70px; float:right" class="hidden next-button btn btn-lg btn-default">Next</button></a>
</div></div>
<a style="margin-top:10px;margin-bottom:10px" target="_blank" href="https://github.com/singularityware/singularityware.github.io/blob/master/pages/_pages/docs/docs-docker.md" class="btn btn-default btn-xs githubEditButton" role="button"><i class="fa fa-github fa-lg"></i> Edit me</a>
<div class="tags">
</div>
</div>
<hr class="shaded"/>
<footer>
<div class="row">
<!-- Social Media links, etc -->
<div class="col-lg-6 footer">
<a class="no-after social-icon" href="https://twitter.com/SingularityApp">
<i class="fa fa-4x fa-twitter no-after"></i>
</a>
<a class="no-after social-icon" href="https://github.com/singularityware">
<i class="fa fa-4x fa-github no-after"></i>
</a>
</div>
<div class="col-lg-6 footer">
<p><img src="images/logo/logo.png" alt="Company logo" style="width:40px;padding-bottom:10px"/></p>
Site last generated: Jul 26, 2017 <br />
</div>
</div>
</footer>
</div>
<!-- /.row -->
</div>
<!-- /.container -->
</div>
</body>
<!-- the google_analytics_id gets auto inserted from the config file -->
<script>(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)})(window,document,'script','//www.google-analytics.com/analytics.js','ga');ga('create','UA-84672381-1','auto');ga('require','displayfeatures');ga('send','pageview');</script>
</html>