Skip to content

Commit 9078f48

Browse files
agibsoncccagibsonccc
committed
Update dl4j versions to stable, remove old repositories used for testing
1 parent 6428a3c commit 9078f48

File tree

22 files changed

+430
-115
lines changed

22 files changed

+430
-115
lines changed

‎README.md

+12-4
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@
1919
The **Eclipse Deeplearning4J** (DL4J) ecosystem is a set of projects intended to support all the needs of a JVM based deep learning application. This means starting with the raw data, loading and preprocessing it from wherever and whatever format it is in to building and tuning a wide variety of simple and complex deep learning networks.
2020

2121
The DL4J stack comprises of:
22-
- **DL4J**: High level API to build MultiLayerNetworks and ComputationGraphs with a variety of layers, including custom ones. Supports importing Keras models from h5, including tf.keras models (as of 1.0.0-beta7) and also supports distributed training on Apache Spark
22+
- **DL4J**: High level API to build MultiLayerNetworks and ComputationGraphs with a variety of layers, including custom ones. Supports importing Keras models from h5, including tf.keras models (as of 1.0.0-M1) and also supports distributed training on Apache Spark
2323
- **ND4J**: General purpose linear algebra library with over 500 mathematical, linear algebra and deep learning operations. ND4J is based on the highly-optimized C++ codebase LibND4J that provides CPU (AVX2/512) and GPU (CUDA) support and acceleration by libraries such as OpenBLAS, OneDNN (MKL-DNN), cuDNN, cuBLAS, etc
2424
- **SameDiff** : Part of the ND4J library, SameDiff is our automatic differentiation / deep learning framework. SameDiff uses a graph-based (define then run) approach, similar to TensorFlow graph mode. Eager graph (TensorFlow 2.x eager/PyTorch) graph execution is planned. SameDiff supports importing TensorFlow frozen model format .pb (protobuf) models. Import for ONNX, TensorFlow SavedModel and Keras models are planned. Deeplearning4j also has full SameDiff support for easily writing custom layers and loss functions.
2525
- **DataVec**: ETL for machine learning data in a wide variety of formats and files (HDFS, Spark, Images, Video, Audio, CSV, Excel etc)
@@ -35,6 +35,17 @@ Users can also refer to the [simple sample project provided](./mvn-project-templ
3535

3636
Build tools are considered standard software engineering best practice. Besides this the complexities posed by the projects in the DL4J ecosystem make dependencies too difficult to manage manually. All the projects in the DL4J ecosystem can be used with other build tools like Gradle, SBT etc. More information on that can be found [here](https://deeplearning4j.konduit.ai/config/buildtools).
3737

38+
## Support
39+
40+
For help with the examples, please go to our [support forum](https://community.konduit.ai/)
41+
42+
Note for users of 1.0.0-beta7 and prior, some examples and modules have been removed to reflect
43+
changes in the framework's direction. Please see and comment on our post [here](https://community.konduit.ai/t/upcoming-removal-of-modules-and-roadmap-changes/1240)
44+
45+
If you would like a workaround for something you may be missing,
46+
please feel free to post on the forums, and we will do what we can to help you.
47+
48+
3849
## Example Content
3950
Projects are based on what functionality the included examples demonstrate to the user and not necessarily which library in the DL4J stack the functionality lives in.
4051

@@ -64,9 +75,6 @@ This project contains a set of examples that demonstrate how raw data in various
6475
- [nd4j-ndarray-examples](nd4j-ndarray-examples/README.md)
6576
This project contains a set of examples that demonstrate how to manipulate NDArrays. The functionality of ND4J demonstrated here can be likened to NumPy.
6677

67-
- [arbiter-examples](arbiter-examples/README.md)
68-
This project contains a set of examples that demonstrate usage of the Arbiter library for hyperparameter tuning of Deeplearning4J neural networks.
69-
7078
- [rl4j-examples](rl4j-examples/README.md)
7179
This project contains examples of using RL4J, the reinforcement learning library in DL4J.
7280

‎android-examples/README.md

-8
Original file line numberDiff line numberDiff line change
@@ -8,14 +8,6 @@ The build configuration for Android can be found in [app/build.gradle](./app/bui
88

99
[Go back](../README.md) to the main repository page to explore other features/functionality of the **Eclipse Deeplearning4J** ecosystem. File an issue [here](https://github.com/eclipse/deeplearning4j-examples/issues) to request new features.
1010

11-
# Known Issues
12-
Due to an unfortunately timed change, the 1.0.0-beta7 release doesn't work well with most Android devices. For this reason, this example project uses the SNAPSHOT version.
13-
14-
Again unfortunately, gradle has issues retrieving SNAPSHOT versions when combined with classifiers, which are used here to retrieve only the android specific backend dependencies.
15-
16-
If you want to run this example locally, you will therefore need to have Maven installed in order to download those specific dependencies.
17-
18-
Once you have installed Maven, you will need to run the following commands from the command line to download the correct android backend files:
1911

2012
```
2113
mvn dependency:get -DremoteRepositories=snapshots::::https://oss.sonatype.org/content/repositories/snapshots -Dartifact=org.nd4j:nd4j-native:1.0.0-SNAPSHOT:jar:android-arm

‎android-examples/app/build.gradle

+1-1
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ android {
2727
}
2828

2929
dependencies {
30-
def dl4jVersion = '1.0.0-SNAPSHOT'
30+
def dl4jVersion = '1.0.0-M1'
3131
def openblasVersion = '0.3.10-1.5.5'
3232
def opencvVersion = '4.5.1-1.5.5'
3333
def leptonicaVersion = '1.80.0-1.5.5'

‎data-pipeline-examples/pom.xml

+31-1
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@
2323

2424
<groupId>org.deeplearning4j</groupId>
2525
<artifactId>data-pipeline-examples</artifactId>
26-
<version>1.0.0-beta7</version>
26+
<version>1.0.0-M1</version>
2727
<name>Building a data pipeline prior to modeling</name>
2828
<description>Loading raw data and processing it before training</description>
2929

@@ -42,6 +42,22 @@
4242
<spark.version>2.4.3</spark.version>
4343
</properties>
4444

45+
<repositories>
46+
<repository>
47+
<id>sonatype-nexus-snapshots</id>
48+
<name>Sonatype Nexus Snapshots</name>
49+
<url>https://oss.sonatype.org/content/repositories/snapshots</url>
50+
<releases>
51+
<enabled>false</enabled>
52+
</releases>
53+
<snapshots>
54+
<enabled>true</enabled>
55+
<updatePolicy>daily</updatePolicy> <!-- Optional, update daily -->
56+
</snapshots>
57+
</repository>
58+
</repositories>
59+
60+
4561
<dependencies>
4662
<dependency>
4763
<groupId>org.nd4j</groupId>
@@ -124,6 +140,20 @@
124140
<!-- Maven Enforcer: Ensures user has an up to date version of Maven before building -->
125141
<build>
126142
<plugins>
143+
144+
<plugin>
145+
<artifactId>maven-surefire-plugin</artifactId>
146+
<version>3.0.0-M5</version>
147+
<inherited>true</inherited>
148+
<dependencies>
149+
<dependency>
150+
<groupId>org.apache.maven.surefire</groupId>
151+
<artifactId>surefire-junit-platform</artifactId>
152+
<version>3.0.0-M5</version>
153+
</dependency>
154+
</dependencies>
155+
</plugin>
156+
127157
<plugin>
128158
<artifactId>maven-enforcer-plugin</artifactId>
129159
<version>1.0.1</version>

‎dl4j-distributed-training-examples/pom.xml

+15-9
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@
2323

2424
<groupId>org.deeplearning4j</groupId>
2525
<artifactId>dl4j-distributed-training-examples</artifactId>
26-
<version>1.0.0-beta7</version>
26+
<version>1.0.0-M1</version>
2727
<name>Introduction to Distributed Training with DL4J</name>
2828
<description>A set of examples introducing distributed training with the DL4J framework</description>
2929

@@ -52,14 +52,6 @@
5252
</properties>
5353

5454
<repositories>
55-
<repository>
56-
<id>orgdeeplearning4j-1165</id>
57-
<url>https://oss.sonatype.org/content/repositories/orgdeeplearning4j-1165</url>
58-
<releases>
59-
<enabled>true</enabled>
60-
</releases>
61-
</repository>
62-
6355
<repository>
6456
<id>sonatype-nexus-snapshots</id>
6557
<name>Sonatype Nexus Snapshots</name>
@@ -75,6 +67,7 @@
7567
</repositories>
7668

7769

70+
7871
<dependencies>
7972
<dependency>
8073
<groupId>org.apache.spark</groupId>
@@ -150,6 +143,19 @@
150143

151144
<build>
152145
<plugins>
146+
<plugin>
147+
<artifactId>maven-surefire-plugin</artifactId>
148+
<version>3.0.0-M5</version>
149+
<inherited>true</inherited>
150+
<dependencies>
151+
<dependency>
152+
<groupId>org.apache.maven.surefire</groupId>
153+
<artifactId>surefire-junit-platform</artifactId>
154+
<version>3.0.0-M5</version>
155+
</dependency>
156+
</dependencies>
157+
</plugin>
158+
153159
<plugin>
154160
<groupId>org.codehaus.mojo</groupId>
155161
<artifactId>exec-maven-plugin</artifactId>

‎dl4j-distributed-training-examples/scripts/patentExamplePreproc.sh

+1-1
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ MINIBATCH=32 #Minibatch size for preprocessed
3232

3333
#Other variables. Don't modify these
3434
SCRIPTDIR=$(dirname "$0")
35-
JARFILE=${SCRIPTDIR}/../target/dl4j-spark-1.0.0-beta7-bin.jar
35+
JARFILE=${SCRIPTDIR}/../target/dl4j-spark-1.0.0-M1-bin.jar
3636
AZURE_ACCT=fs.azure.account.key.${AZURE_STORAGE_ACCT}.blob.core.windows.net
3737

3838

‎dl4j-distributed-training-examples/scripts/patentExampleTrain.sh

+1-1
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ AERON_BUFFER=33554432
4040

4141
#Other variables. Don't modify these
4242
SCRIPTDIR=$(dirname "$0")
43-
JARFILE=${SCRIPTDIR}/../target/dl4j-spark-1.0.0-beta7-bin.jar
43+
JARFILE=${SCRIPTDIR}/../target/dl4j-spark-1.0.0-M1-bin.jar
4444
AZURE_ACCT=fs.azure.account.key.${AZURE_STORAGE_ACCT}.blob.core.windows.net
4545

4646

‎dl4j-distributed-training-examples/scripts/tinyImagenetTrain.sh

+1-1
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ EXECUTOR_MEMORY=12G
3232
MASTER_PORT=7077 # Port for the spark master. Default is 7077
3333
MINIBATCH=32 # Minibatch size for preprocessed datasets
3434
SCRIPTDIR=$(dirname "$0")
35-
JARFILE=${SCRIPTDIR}/../target/dl4j-spark-1.0.0-beta7-bin.jar
35+
JARFILE=${SCRIPTDIR}/../target/dl4j-spark-1.0.0-M1-bin.jar
3636

3737
#Memory
3838
OFFHEAP_MEM_JAVACPP=20G # Maximum amount of off-heap memory

‎dl4j-examples/README.md

+2
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@ The pom file in this project can be used as a template for a user's own project.
66

77
[Go back](../README.md) to the main repository page to explore other features/functionality of the **Eclipse Deeplearning4J** ecosystem. File an issue [here](https://github.com/eclipse/deeplearning4j-examples/issues) to request new features.
88

9+
10+
911
### QUICKSTART
1012

1113
#### Modeling Examples

‎dl4j-examples/pom.xml

+17-15
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@
3131
<dl4j-master.version>1.0.0-M1</dl4j-master.version>
3232
<!-- Change the nd4j.backend property to nd4j-cuda-X-platform to use CUDA GPUs -->
3333
<!-- <nd4j.backend>nd4j-cuda-10.2-platform</nd4j.backend> -->
34-
<nd4j.backend>nd4j-native</nd4j.backend>
34+
<nd4j.backend>nd4j-cuda-11.2</nd4j.backend>
3535
<java.version>1.8</java.version>
3636
<maven-compiler-plugin.version>3.8.1</maven-compiler-plugin.version>
3737
<maven.minimum.version>3.3.1</maven.minimum.version>
@@ -48,15 +48,6 @@
4848

4949

5050
<repositories>
51-
<repository>
52-
<id>orgdeeplearning4j-1165</id>
53-
<url>https://oss.sonatype.org/content/repositories/orgdeeplearning4j-1165</url>
54-
<releases>
55-
<enabled>true</enabled>
56-
</releases>
57-
</repository>
58-
59-
6051
<repository>
6152
<id>sonatype-nexus-snapshots</id>
6253
<name>Sonatype Nexus Snapshots</name>
@@ -77,6 +68,8 @@
7768
<artifactId>${nd4j.backend}</artifactId>
7869
<version>${dl4j-master.version}</version>
7970
</dependency>
71+
72+
8073
<dependency>
8174
<groupId>org.datavec</groupId>
8275
<artifactId>datavec-api</artifactId>
@@ -162,17 +155,26 @@
162155
<scope>test</scope>
163156
</dependency>
164157

165-
<dependency>
166-
<groupId>org.bytedeco</groupId>
167-
<artifactId>cuda-platform-redist</artifactId>
168-
<version>11.2-8.1-1.5.5</version>
169-
</dependency>
170158

171159

172160
</dependencies>
173161
<!-- Maven Enforcer: Ensures user has an up to date version of Maven before building -->
174162
<build>
175163
<plugins>
164+
165+
<plugin>
166+
<artifactId>maven-surefire-plugin</artifactId>
167+
<version>3.0.0-M5</version>
168+
<inherited>true</inherited>
169+
<dependencies>
170+
<dependency>
171+
<groupId>org.apache.maven.surefire</groupId>
172+
<artifactId>surefire-junit-platform</artifactId>
173+
<version>3.0.0-M5</version>
174+
</dependency>
175+
</dependencies>
176+
</plugin>
177+
176178
<plugin>
177179
<artifactId>maven-enforcer-plugin</artifactId>
178180
<version>1.0.1</version>

‎dl4j-examples/src/test/resources/log4j.properties

+42
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
################################################################################
2+
# Copyright (c) 2020 Konduit K.K.
3+
# Copyright (c) 2015-2019 Skymind, Inc.
4+
#
5+
# This program and the accompanying materials are made available under the
6+
# terms of the Apache License, Version 2.0 which is available at
7+
# https://www.apache.org/licenses/LICENSE-2.0.
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
11+
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
12+
# License for the specific language governing permissions and limitations
13+
# under the License.
14+
#
15+
# SPDX-License-Identifier: Apache-2.0
16+
################################################################################
17+
18+
19+
log4j.rootLogger=ERROR, Console
20+
log4j.logger.play=DEBUG
21+
log4j.appender.Console=org.apache.log4j.ConsoleAppender
22+
log4j.appender.Console.layout=org.apache.log4j.PatternLayout
23+
log4j.appender.Console.layout.ConversionPattern=%d{ABSOLUTE} %-5p ~ %m%n
24+
25+
log4j.appender.org.springframework=DEBUG
26+
log4j.appender.org.nd4j=INFO
27+
log4j.appender.org.canova=INFO
28+
log4j.appender.org.datavec=INFO
29+
log4j.appender.org.deeplearning4j=INFO
30+
log4j.appender.opennlp.uima=OFF
31+
log4j.appender.org.apache.uima=OFF
32+
log4j.appender.org.cleartk=OFF
33+
34+
log4j.logger.org.springframework=INFO
35+
log4j.logger.org.nd4j=INFO
36+
log4j.logger.org.canova=INFO
37+
log4j.logger.org.datavec=INFO
38+
log4j.logger.org.deeplearning4j=INFO
39+
log4j.logger.opennlp.uima.util=OFF
40+
log4j.logger.org.apache.uima=OFF
41+
log4j.logger.org.cleartk=OFF
42+

‎dl4j-examples/src/test/resources/logback.xml

+54
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
<!--~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2+
~ Copyright (c) 2020 Konduit K.K.
3+
~ Copyright (c) 2015-2019 Skymind, Inc.
4+
~
5+
~ This program and the accompanying materials are made available under the
6+
~ terms of the Apache License, Version 2.0 which is available at
7+
~ https://www.apache.org/licenses/LICENSE-2.0.
8+
~
9+
~ Unless required by applicable law or agreed to in writing, software
10+
~ distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
11+
~ WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
12+
~ License for the specific language governing permissions and limitations
13+
~ under the License.
14+
~
15+
~ SPDX-License-Identifier: Apache-2.0
16+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~-->
17+
18+
<configuration>
19+
20+
21+
22+
<appender name="FILE" class="ch.qos.logback.core.FileAppender">
23+
<file>logs/application.log</file>
24+
<encoder>
25+
<pattern>%date - [%level] - from %logger in %thread
26+
%n%message%n%xException%n</pattern>
27+
</encoder>
28+
</appender>
29+
30+
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
31+
<encoder>
32+
<pattern> %logger{15} - %message%n%xException{5}
33+
</pattern>
34+
</encoder>
35+
</appender>
36+
37+
<logger name="org.apache.catalina.core" level="DEBUG" />
38+
<logger name="org.springframework" level="DEBUG" />
39+
<logger name="org.deeplearning4j" level="INFO" />
40+
<logger name="org.canova" level="INFO" />
41+
<logger name="org.datavec" level="INFO" />
42+
<logger name="org.nd4j" level="INFO" />
43+
<logger name="opennlp.uima.util" level="OFF" />
44+
<logger name="org.apache.uima" level="OFF" />
45+
<logger name="org.cleartk" level="OFF" />
46+
47+
48+
49+
<root level="ERROR">
50+
<appender-ref ref="STDOUT" />
51+
<appender-ref ref="FILE" />
52+
</root>
53+
54+
</configuration>

0 commit comments

Comments
 (0)