Bigtop Project Release Notes

Release Notes - Bigtop - Version 0.8.0

Sub-task

  • [BIGTOP-1111] - Bump version of Hadoop to 2.2.0
  • [BIGTOP-1165] - Ivy config for hive metastore looks incorrect, which breaks build on JDK 7
  • [BIGTOP-1166] - Hive-jdbc doesn't compile on JDK 7, which breaks build
  • [BIGTOP-1183] - Rename spark package to spark-core
  • [BIGTOP-1184] - bump hadoop version to 2.3.0
  • [BIGTOP-1202] - One-button-click way for installing all needed bits of testing framework
  • [BIGTOP-1203] - Update toolchaing to automatically install Gradle into dev env.
  • [BIGTOP-1208] - Add DFSIO test into longevity suite
  • [BIGTOP-1209] - Add SLive test into longevity suite
  • [BIGTOP-1242] - bump jsvc to 1.0.15
  • [BIGTOP-1244] - bump flume to 1.5.0.1
  • [BIGTOP-1278] - bump pig version to 0.12.1
  • [BIGTOP-1279] - bump hive version to 0.12.0
  • [BIGTOP-1280] - bump Oozie version to 4.0.1
  • [BIGTOP-1281] - bump hbase version to 0.98.2
  • [BIGTOP-1282] - bump Mahout version to 0.9
  • [BIGTOP-1283] - bump Solr version to 4.6.0
  • [BIGTOP-1284] - bump Spark version to 0.9.1
  • [BIGTOP-1285] - Bump version of Phoenix to 4.0
  • [BIGTOP-1291] - On a clean ~/.m2 hbase mvn install site will fail as install is executed first
  • [BIGTOP-1293] - Include Hive 0.13 into the stack
  • [BIGTOP-1302] - Update hbase smoke tests to escape Hbase's "event horizon"
  • [BIGTOP-1330] - bump Giraph version to 1.1.0
  • [BIGTOP-1331] - bump Hue version to 3.6.0
  • [BIGTOP-1340] - Hbase build has to use hadoop-two.version sys. prop
  • [BIGTOP-1350] - Bump version of Crunch to 0.10.0
  • [BIGTOP-1354] - bump version of Solr to 4.9.0
  • [BIGTOP-1426] - Bump version of Hadoop to 2.4.1
  • [BIGTOP-1432] - Update HBase version to 0.98.5 in the BOM
  • [BIGTOP-1433] - Fix Hive build after BIGTOP-1429
  • [BIGTOP-1434] - Maven build failing with NoClassDefFoundError at maven-site-plugin:3.0-beta-3:site (packaging-documentation) @ sqoop-docs
  • [BIGTOP-1435] - Fix Crunch build after BIGTOP-1429

Bug

  • [BIGTOP-282] - the licensing status of the MovieLens data files needs to be cleared up
  • [BIGTOP-831] - Hadoop branch-2 (and trunk?) are failing when built with JDK7u11
  • [BIGTOP-922] - Existing Puppet recipes are failing to initialize HDFS dictories on the first run
  • [BIGTOP-986] - install-flume.sh uses -x instead of -e
  • [BIGTOP-996] - Document steps needed to run BT tests on a fresh host
  • [BIGTOP-997] - TestCLI fails test id 5: comparator fails
  • [BIGTOP-1042] - Add support for Hadoop 2.1.x-beta
  • [BIGTOP-1048] - ZooKeeper installation use "zookeeper-server.pid" as default while ZooKeeper expects zookeeper_server.pid
  • [BIGTOP-1072] - Vagrant scripts for spinning up and "hydrating" bigtop vms
  • [BIGTOP-1092] - Add Hue documentation package
  • [BIGTOP-1097] - introduce bigtop-groovy package
  • [BIGTOP-1124] - solrctl should be able to auto-detect JAVA_HOME
  • [BIGTOP-1125] - Return value does not reflect status checks
  • [BIGTOP-1128] - FIX and modularize mahout sample data sets
  • [BIGTOP-1132] - Phoenix RPM specfile does not create symlinks
  • [BIGTOP-1141] - Add TestYarnNode to test yarn node
  • [BIGTOP-1143] - Oozie configuration file is incomplete
  • [BIGTOP-1146] - Flume NG package is broken because of org.hbase.asynchbase deps. mismanagement
  • [BIGTOP-1150] - update Spark debian packaging to the latest format
  • [BIGTOP-1153] - Crunch package doesn't depend on anything
  • [BIGTOP-1164] - Get whole Bigtop stack building on JDK 7
  • [BIGTOP-1167] - got "Permission denied" when creating vagrant home folder in provision.sh
  • [BIGTOP-1168] - hue build fails (unresolvable maven dependencies)
  • [BIGTOP-1169] - Hue rpm build fails
  • [BIGTOP-1174] - missing dependency setting on init-hdfs.sh in puppet recipes
  • [BIGTOP-1181] - Add pyspark to spark package
  • [BIGTOP-1187] - Spark service packages have extraneous core in the name
  • [BIGTOP-1191] - master pom.xml file formatting is out of whack
  • [BIGTOP-1197] - iTest PackageManagerTests are failing
  • [BIGTOP-1199] - Makefile for deb packing needs to support zip archives
  • [BIGTOP-1205] - GRADLE_OPTS missing closing " in toolchain jenkins.sh.*
  • [BIGTOP-1217] - Spark test-artifact missing repository for install
  • [BIGTOP-1218] - Add JDK7 to Bigtop toolchain
  • [BIGTOP-1220] - Gradle task order isn't enforced properly with dependsOn
  • [BIGTOP-1221] - Expand and updated FUSE tests
  • [BIGTOP-1223] - TestSlive does not fully remove root directory after an iteration
  • [BIGTOP-1225] - TestHDFSQuota can fail if tests are run out of order
  • [BIGTOP-1226] - TestSLive setup() already being used, need to rename
  • [BIGTOP-1232] - test artifact build fails in io-longevity
  • [BIGTOP-1234] - TestFileAppend shouldn't reuse client during testMultipleOutputStreamFailure()
  • [BIGTOP-1236] - TestCLI expecting permissions incorrectly
  • [BIGTOP-1238] - TestCLI has misspelled 'NAMNEODE' that is used without protocol name
  • [BIGTOP-1239] - TestCLI: make ports optional in regex comparators
  • [BIGTOP-1246] - Fix the shell-object bug in HttpFs smoke tests
  • [BIGTOP-1250] - TestTextSnappy fails
  • [BIGTOP-1252] - Update ant from 1.9.2 to 1.9.3
  • [BIGTOP-1253] - TestSLive calling setpDir() instead of setupDir()
  • [BIGTOP-1254] - add gradle specific files to .gitignore
  • [BIGTOP-1255] - init-hcfs.json needs to encode privileges as strings, not numbers.
  • [BIGTOP-1256] - Spark package deployment should reflect BIGTOP-1183
  • [BIGTOP-1258] - Spark puppet deploy might fail as /etc/spark can not be created
  • [BIGTOP-1259] - spark-worker service should be started after spark-master if ran on the same node
  • [BIGTOP-1260] - Puppet deployment should reflect on the new name of spark package
  • [BIGTOP-1268] - Jenkins Job Bigtop-trunk-Hadoop failed from 3/15/2014 due to protoc version
  • [BIGTOP-1274] - yarn ResourceManager should be started before starting NodeManagers
  • [BIGTOP-1290] - Zookeeper build is failing on Ant 1.9.3
  • [BIGTOP-1294] - asciidoc is missed from centos and sles environments of bigtop_toolchain
  • [BIGTOP-1295] - package.mk misteriously missing mkdir for RPMS that leads to SUSE build to fail
  • [BIGTOP-1300] - need to bump ant version to 1.9.4 in bigtop_toolchain
  • [BIGTOP-1303] - Pig 0.12.1 build is broken at the site target. Stack build can't proceed
  • [BIGTOP-1304] - bigtop_toolchain needs to install Scala 2.10 to support Spark build
  • [BIGTOP-1306] - RPM licensing metadata is inconsistent and wrong
  • [BIGTOP-1313] - Ant 1.9.4 update is missed from ant.pp recipe.
  • [BIGTOP-1319] - hive-rpm target is broken
  • [BIGTOP-1322] - hbase_install.sh shall be called with bash
  • [BIGTOP-1324] - Fix typo in testHDFSConf.xml of TestCLI
  • [BIGTOP-1336] - Puppet recipes failed to deploy kerberos enabled hadoop cluster
  • [BIGTOP-1337] - Logging success messages in service startup and shutdown only after verification.
  • [BIGTOP-1345] - remove bigtop-packages/src/root@mgrover-centos6-2.ent.cloudera.com
  • [BIGTOP-1346] - Generalize evaluation regexp in packages.gradle
  • [BIGTOP-1355] - Hive 0.13 deb build is broken
  • [BIGTOP-1358] - adjust toolchain to use JDK7 by default
  • [BIGTOP-1359] - fix a few issues with our puppet_toolchain code
  • [BIGTOP-1371] - Spark-master service fails to start due to missing /usr/lib/spark/work
  • [BIGTOP-1380] - Pig might run out of memory during site construction
  • [BIGTOP-1382] - Hadoop site.csv components additions: yarn & mapred-app
  • [BIGTOP-1390] - JDK Woes
  • [BIGTOP-1394] - package test manifest got out of date
  • [BIGTOP-1396] - update metadata for bigtop-groovy and phoenix
  • [BIGTOP-1397] - hive-hbase package is missing after move to the latest Hive
  • [BIGTOP-1398] - hadoop-hdfs-fuse can't be installed on Ubuntu 14.04
  • [BIGTOP-1399] - add --help to alternative invocation to support newer Ubuntus
  • [BIGTOP-1400] - Whirr is failing with PermGen error: needs to be bumped up
  • [BIGTOP-1401] - Create man pages for YARN, HDFS and MAPRED.
  • [BIGTOP-1402] - Update bigtop gradle to 2.x
  • [BIGTOP-1405] - Gradle 2.0 installation from toolchain is broken
  • [BIGTOP-1415] - Remove redundant jars from spark package.
  • [BIGTOP-1418] - [BigPetStore] build.gradle - wrong variable name (hadoopClusterRuntime instead of runtime)
  • [BIGTOP-1420] - Update Phoenix version to 4.1.0 in the BOM
  • [BIGTOP-1421] - Phoenix build is missing some variables
  • [BIGTOP-1425] - gradle build can fail with NPE in an obscure way
  • [BIGTOP-1427] - HBase build should build hbase-***-hadoop2 by default
  • [BIGTOP-1428] - update tests to reflect new home for hbase user
  • [BIGTOP-1429] - Apache build of Bigtop test artifacts fails to find correct Hbase jars
  • [BIGTOP-1430] - incorrect version of spark-core prevents test artifacts from compiling
  • [BIGTOP-1438] - asciidoc is no longer needed by Sqoop
  • [BIGTOP-1439] - split jdk7 and jdk6 in Bigtop toolchain
  • [BIGTOP-1440] - a few usability updates to bigtop_toolchain
  • [BIGTOP-1441] - bigtop_toolchain fails on Ubuntu
  • [BIGTOP-1442] - need to add proper clean up for the toolchain puppet code
  • [BIGTOP-1444] - Phoenix RPM build is broken after BIGTOP-1421

Improvement

  • [BIGTOP-840] - Refactor Bigtop puppet code for better integration with newer Puppet and Whirr
  • [BIGTOP-848] - Allow to build stack on top of an arbitrary Hadoop SHA
  • [BIGTOP-952] - init-hdfs.sh is dog slow. Let's replace it with a direct HDFS API calls and better layout management
  • [BIGTOP-992] - Create HowToContribute wiki page
  • [BIGTOP-1019] - Remove mysql requirement constraint from sqoop tests
  • [BIGTOP-1026] - Big Top Mapred/Itest assertions should report specific commands which fail
  • [BIGTOP-1029] - Add support for HBase 0.96+
  • [BIGTOP-1085] - spark packages needs to be split
  • [BIGTOP-1110] - Define BOM for 0.8.0 release of Bigtop
  • [BIGTOP-1130] - Update Build Requirements on our web pages
  • [BIGTOP-1137] - provide a way to request a major version of JDK
  • [BIGTOP-1138] - hadoop-conf-pseudo package uses the deprecated fs.default.name parameter
  • [BIGTOP-1139] - bigtop-tomcat and bigtop-jsvc don't depend on bigtop-utils
  • [BIGTOP-1144] - Need to update command 'puppet apply' in file bigtop-deploy/puppet/README.md
  • [BIGTOP-1147] - README.md improvments
  • [BIGTOP-1152] - speed up spark build
  • [BIGTOP-1155] - New logo !
  • [BIGTOP-1161] - Have a way to dynamically select components to be deployed by Puppet recipes.
  • [BIGTOP-1163] - puppet apply shows db init error if the oozie server is already running
  • [BIGTOP-1171] - Puppetize the Vagrant deployer
  • [BIGTOP-1178] - Clusterize the puppetized vagrant deployer.
  • [BIGTOP-1195] - Higher level interface to smokes.
  • [BIGTOP-1200] - Implement Generic Text File to define HCFS filesystem semantics
  • [BIGTOP-1201] - Enhance (gradleize) the build to ease development, deployment; abstract implementation
  • [BIGTOP-1207] - Create a foundation layer to build longevity and performance tests for the stack
  • [BIGTOP-1210] - integrate BIGTOP-952 with plain text HCFS FS definition from BIGTOP-1200
  • [BIGTOP-1213] - TestHadoopExamples smokes: orderering of tests; other improvements
  • [BIGTOP-1222] - Simplify and gradleize a subset of the bigtop smokes
  • [BIGTOP-1224] - Provide a simple order for tests
  • [BIGTOP-1241] - BIGTOP should update protobuf to 2.5 from 2.4.x
  • [BIGTOP-1257] - Make TestHadoopExamples modular enough to support longevity tests.
  • [BIGTOP-1269] - BigPetStore: Create build w/ gradle
  • [BIGTOP-1289] - Update component versions in the pom.xml (including hadoop)
  • [BIGTOP-1296] - Fix permission and Update README for vagrant-puppet recipe
  • [BIGTOP-1311] - allow easy site publishing
  • [BIGTOP-1312] - Add dynamic menus, update, links, modern CSS to bigtop site.
  • [BIGTOP-1314] - Deprecate make build
  • [BIGTOP-1316] - enhance Shell for better checking of return code & output logging
  • [BIGTOP-1321] - TestCLI: check assumptions before the test run
  • [BIGTOP-1342] - Make TestCLI usable for both HDFS and HCFS
  • [BIGTOP-1347] - Support better entropy performance on vagrant VMs
  • [BIGTOP-1348] - vagrant-puppet provisioning is too slow on installing packages
  • [BIGTOP-1372] - Bigtop needs feature that takes in multiple arguments to build specific components at command line
  • [BIGTOP-1374] - Add dynamic override of BOM variables into Gradle build
  • [BIGTOP-1391] - Improve README.md so that the markdown rendering is pretty and easy to read

New Feature

  • [BIGTOP-1039] - Add packaging for Shark analytics
  • [BIGTOP-1089] - BigPetStore: A polyglot big data processing blueprint
  • [BIGTOP-1188] - Smoke tests for HttpFs
  • [BIGTOP-1192] - Add utilities to facilitate cluster failure testing into bigtop-test-framework
  • [BIGTOP-1212] - Pick or build a framework for building fake data sets
  • [BIGTOP-1272] - BigPetStore: Productionize the Mahout recommender
  • [BIGTOP-1351] - Init script for HBase Thrift2 server is missing

Task

  • [BIGTOP-236] - The user installed by hbase gets an incorrect home directory
  • [BIGTOP-1240] - Formatting Guidelines

Test

  • [BIGTOP-1040] - Add integration tests for Shark component

Release Notes - Bigtop - Version 0.7.0

Sub-task

Bug

  • [ BIGTOP-990 ] - /usr/lib/bigtop-utils/ should be owned by bigtop-utils package
  • [ BIGTOP-998 ] - Consolidate Hue packages
  • [ BIGTOP-1005 ] - Create versionless symlinks in hadoop client directory
  • [ BIGTOP-1006 ] - Explicitly state the working directory to be used in init scripts instead of relying on user's home directories
  • [ BIGTOP-1008 ] - /usr/lib/bigtop-utils directory is not owned by bigtop-utils package in RPMs
  • [ BIGTOP-1010 ] - libsnappy not included in the final hadoop RPM
  • [ BIGTOP-1011 ] - bigtop-detect-javahome has a quirky search order
  • [ BIGTOP-1017 ] - Revert a commit that introduces some unneeded dependencies
  • [ BIGTOP-1027 ] - 'bigtop-detect-classpath' helper script missing from bigtop-utils
  • [ BIGTOP-1033 ] - itest-common: fix debian install to avoid conffile prompt
  • [ BIGTOP-1034 ] - Custom user's home directory's creation fails in init-hdfs.sh
  • [ BIGTOP-1036 ] - Hue's oozie app package should depend on hue jobsub package
  • [ BIGTOP-1037 ] - Provide a mechanism to control the sourcing of defaults files
  • [ BIGTOP-1044 ] - hive and hive-jdbc versions should be kept in sync
  • [ BIGTOP-1045 ] - Be consistent with shells, bigtop-detect-javahome, and bigtop-utils versions
  • [ BIGTOP-1046 ] - Pig's use of BIGTOP_DEFAULTS_DIR is evaluated at the wrong time
  • [ BIGTOP-1051 ] - Spark archive should come in tar.gz format
  • [ BIGTOP-1052 ] - Increase environment configurability/debugging of Mahout Tests
  • [ BIGTOP-1055 ] - Spark package doesn't create PREFIX/bin directory and sets wrong permissions
  • [ BIGTOP-1059 ] - Add path option for pig.jar to pig smoke pom.xml
  • [ BIGTOP-1070 ] - Add BIGTOP_CLASSPATH functionality to tomcat deployment
  • [ BIGTOP-1071 ] - bigtop-detect-javahome should select the first possible candidate in the list
  • [ BIGTOP-1081 ] - deb spark packing needs to set SOURCE_DIR
  • [ BIGTOP-1082 ] - spark package tests are missing
  • [ BIGTOP-1084 ] - spark build on CentOS 5 and SLES11 is broken
  • [ BIGTOP-1086 ] - fix miscellaneous failures in package tests
  • [ BIGTOP-1087 ] - flume rpm package needs to own /usr/lib/flume/tools
  • [ BIGTOP-1088 ] - oozie packages need to own /etc/oozie/conf.dist/tomcat-deployment.{default,secure}
  • [ BIGTOP-1090 ] - hue packaging needs to handle mutable bits (app.reg and hue.pth)
  • [ BIGTOP-1094 ] - spark SPEC mistakenly re-defines alternatives command on SLES/OpenSUSE
  • [ BIGTOP-1095 ] - oozie packaging doesn't remove tomcat alternatives
  • [ BIGTOP-1096 ] - alternatives within the alternatives-managed sub-directory could be harmful
  • [ BIGTOP-1098 ] - now that Spark has had its first incubator release we have to adjust bigtop accordingly
  • [ BIGTOP-1099 ] - sync up the versions used for smoke tests with the BOM versions
  • [ BIGTOP-1100 ] - the new Tomcat deployment style would benefit from a few usability tweaks
  • [ BIGTOP-1103 ] - Remaining fixes of BIGTOP_DEFAULTS_DIR evaluation
  • [ BIGTOP-1107 ] - Update to the official Hue 2.5.1 release
  • [ BIGTOP-1108 ] - Solr 4.5.0 has been formally released and we should start using the official release bits
  • [ BIGTOP-1112 ] - sqoop-client should require bigtop-utils
  • [ BIGTOP-1114 ] - Hue metastore app is missing from core registration and Debian packages
  • [ BIGTOP-1115 ] - update package test manifest for Bigtop 0.7.0
  • [ BIGTOP-1117 ] - hive-hcatalog integration has a few issues
  • [ BIGTOP-1119 ] - phoenix DEB packages differ in a few ways from RPM counterparts

Improvement

  • [ BIGTOP-927 ] - Update puppet recipes to support Ubuntu
  • [ BIGTOP-938 ] - Upgrade to Junit 4.11
  • [ BIGTOP-955 ] - HBase installation should advertise its location and configuration
  • [ BIGTOP-1004 ] - Hadoop packages do not include version independent symlinks
  • [ BIGTOP-1021 ] - Update HBase version to 0.94.9
  • [ BIGTOP-1031 ] - README has outdated/ambiguous information
  • [ BIGTOP-1041 ] - Add Puppet support for Spark cluster deployment
  • [ BIGTOP-1056 ] - some changes are introduced into Spark build: needs to be updated here
  • [ BIGTOP-1063 ] - Add Scala 2.9.3 to bigtop-toolchain
  • [ BIGTOP-1069 ] - Remove descriptions about JDK5 dependency
  • [ BIGTOP-1080 ] - Change /usr/bin scripts to be alternatives instead of flat files
  • [ BIGTOP-1106 ] - update our puppet deployment topology

New Feature

  • [ BIGTOP-811 ] - Add /var/lib/bigtop as a location to install SQL connectors and other plug-ins
  • [ BIGTOP-1023 ] - Define BOM for 0.7.0 release of Bigtop

Task

Test

  • [ BIGTOP-719 ] - Add TestTextSnappy to test hadoop fs -text with snappy compressed files
  • [ BIGTOP-1030 ] - Develop integration tests for new Spark component
  • [ BIGTOP-1057 ] - Add TeraGen / TeraSort Benchmakring

Release Notes - Bigtop - Version 0.6.0

Sub-task

  • [ BIGTOP-239 ] - HBase init scripts are shut down in the wrong order
  • [ BIGTOP-812 ] - HttpFS is using Tomcat 6.0.32 which has numerous vulnerabilities
  • [ BIGTOP-813 ] - add integration test for HCatlog
  • [ BIGTOP-814 ] - add packaging test for HCatlog
  • [ BIGTOP-822 ] - Bump version of Hadoop to 2.0.5
  • [ BIGTOP-824 ] - Bump version of Hive to 0.10
  • [ BIGTOP-832 ] - Bump version of Hue to 2.3.0
  • [ BIGTOP-833 ] - Update HCatalog packaging for Bigtop 0.6.0 integration
  • [ BIGTOP-839 ] - add packaging test for Sqoop 2
  • [ BIGTOP-852 ] - Improve HDFS helper script
  • [ BIGTOP-854 ] - add package testing manifest for HCatalog
  • [ BIGTOP-855 ] - add integration tests for HCatalog
  • [ BIGTOP-861 ] - init-hdfs.sh does not work from init scripts
  • [ BIGTOP-862 ] - All the Apache Hadoop services ought to start nicely and in order
  • [ BIGTOP-863 ] - verify the existence of the initscripts with typo
  • [ BIGTOP-864 ] - update the appliance for the location of the init-hdfs.sh script
  • [ BIGTOP-880 ] - Bump version of Datafu to 0.0.6
  • [ BIGTOP-891 ] - TestPigTest doesn't run against YARN cluster
  • [ BIGTOP-915 ] - bump version of Oozie to 3.3.2
  • [ BIGTOP-916 ] - Bump version of Pig to 0.11.1
  • [ BIGTOP-917 ] - Bump version of Solr to 4.2.1
  • [ BIGTOP-926 ] - Bump version of Giraph to 1.0
  • [ BIGTOP-933 ] - Need to add ASF license text to bigtop-toolchain files
  • [ BIGTOP-962 ] - Bump Sqoop version to 1.99.2
  • [ BIGTOP-963 ] - bump version of Whirr to 0.8.2
  • [ BIGTOP-980 ] - bigtop-toolchaing user module can't create 'jenkins' user

Bug

  • [ BIGTOP-275 ] - scoping of variable in puppet code is incorrect
  • [ BIGTOP-379 ] - Package testing for Bigtop 0.5.0 release
  • [ BIGTOP-445 ] - yarn default file has the wrong name
  • [ BIGTOP-463 ] - should we reconsider /usr/lib vs. /usr/libexec decision?
  • [ BIGTOP-483 ] - Smoke test of Hadoop fails with clitest missing testConf.xml file
  • [ BIGTOP-489 ] - TestCLI with two test cases fails ID - 146 and 147 related to refreshServiceAcl
  • [ BIGTOP-504 ] - provide a functionality for detecting libjvm.so in bigtop-utils
  • [ BIGTOP-545 ] - package testing manifest in trunk needs to be updated
  • [ BIGTOP-553 ] - Metadata for packages needs to be harmonized between RPM and Debain
  • [ BIGTOP-584 ] - Fix hive smoke test failures in trunk
  • [ BIGTOP-624 ] - running start on a service already running should return success
  • [ BIGTOP-637 ] - Update boxgrinder appliance for the coming release
  • [ BIGTOP-786 ] - Not able to find wiki link on Bigtop homepage
  • [ BIGTOP-804 ] - org.apache.bigtop.itest.pmanager.PackageManagerTest.testLookupGcc fails on some systems
  • [ BIGTOP-808 ] - hadoop-conf-pseudo package breaks hadoop-conf alternatives during upgrades
  • [ BIGTOP-809 ] - Remove harmless scary message from solr build
  • [ BIGTOP-815 ] - Move the trunk forward now that 0.5.0 is released
  • [ BIGTOP-818 ] - Updated list of supported platforms on Bigtop's homepage and readme file
  • [ BIGTOP-823 ] - Remove obsolete slf4j removal in hive packaging
  • [ BIGTOP-826 ] - Wrong return code returned when stopping Hue on RHEL/SUSE
  • [ BIGTOP-827 ] - Ubuntu Install Instructutions Fail with a 404
  • [ BIGTOP-829 ] - Hue status returned wrong information on SLES
  • [ BIGTOP-830 ] - Add $(APACHE_MIRROR) for bigtop-tomcat
  • [ BIGTOP-834 ] - Remove workaround for MAPREDUCE-4814 in historyserver
  • [ BIGTOP-835 ] - The shell exec method must have variants which have timeout and can run in background
  • [ BIGTOP-836 ] - Hue status on SuSE doesn't work when run from outside of /usr/bin
  • [ BIGTOP-837 ] - Oozie 3.3.0 can't be build against Hadoop 2.0.3 or later
  • [ BIGTOP-841 ] - hadoop-conf-pseudo missed configuration for capacity scheduler and historyserver
  • [ BIGTOP-842 ] - hadoop-mapreduce-historyserver service dumps some output to the console instead of its .out file
  • [ BIGTOP-844 ] - hadoop rpm upgrade sequence is broken
  • [ BIGTOP-846 ] - The helper script introduced in BIGTOP-547 doesn't create /user/$USER directories on HDFS
  • [ BIGTOP-850 ] - Remove non-ASF repositories and infrastructure references from the project files
  • [ BIGTOP-851 ] - Typo breaks detection of multiple libraries in bigtop-utils in some JVMs
  • [ BIGTOP-853 ] - HBase test, TestLoadAndVerify does not work on secure clusters
  • [ BIGTOP-857 ] - Inconsistent group ownership of /var/run/hadoop-hdfs between .deb and .rpm installs
  • [ BIGTOP-858 ] - HBase tests are no longer compile with HBase 0.94
  • [ BIGTOP-859 ] - fix the Bigtop trunk package test
  • [ BIGTOP-869 ] - hadoop preun should not stop services itself on behalf of its subpackages
  • [ BIGTOP-870 ] - Bump version of Pig to 0.11
  • [ BIGTOP-873 ] - Hive needs Ivy cache cleared
  • [ BIGTOP-874 ] - make test contracts show up in javadoc
  • [ BIGTOP-875 ] - hadoop-smoke is broken because of missed dependency
  • [ BIGTOP-876 ] - TestDFSAdmin in hadoop-smoke is outdated and still relies on presence of -upgradeProgress
  • [ BIGTOP-877 ] - TestDFSAdmin in hadoop-smoke has to ensure its own environment, e.g. create HDFS paths, etc.
  • [ BIGTOP-878 ] - Bump version of Oozie to 3.3.1
  • [ BIGTOP-879 ] - Bump version of Solr to 4.2.0
  • [ BIGTOP-883 ] - Add support for Oracle RPM Java 1.7
  • [ BIGTOP-884 ] - hive smoke tests can't be executed as a jar file isn't present anymore
  • [ BIGTOP-885 ] - TestHiveSmokeBulk fails on Hive 0.9
  • [ BIGTOP-886 ] - Non-fatal errors when starting Hive daemons in EL5
  • [ BIGTOP-887 ] - Sqoop 2 should be consistent with Oozie / Bigtop for server plugins directory
  • [ BIGTOP-888 ] - Oozie losing value of JAVA_HOME when spawning sub-shell
  • [ BIGTOP-889 ] - pig smoke tests are no longer running
  • [ BIGTOP-890 ] - Remove workaround for distributed cache problem from Oozie package
  • [ BIGTOP-892 ] - hadoop packages misses capacity-scheduler configuration
  • [ BIGTOP-894 ] - Pig compilation fails on RPM systems on Bigtop trunk
  • [ BIGTOP-895 ] - A number of testcases in TestCLI are failing with (at least) Hadoop 2.0.3 and later
  • [ BIGTOP-896 ] - Solr rpm requires redhat-lsb to run
  • [ BIGTOP-899 ] - enabled deployment topology with 3 nodes
  • [ BIGTOP-900 ] - update deployment configs for Kerberized installations
  • [ BIGTOP-901 ] - Oozie smoke tests mightn't be able to obtain the jobtracker or namenode hostnames, using old config. names
  • [ BIGTOP-902 ] - Hive uses same file for log4j and standard output
  • [ BIGTOP-903 ] - crunch integration tests need to clean up output directory before each individual subtask is run
  • [ BIGTOP-904 ] - mahout tests should default to /tmp as a temp location
  • [ BIGTOP-905 ] - bump versions of test execution poms to match the BOM
  • [ BIGTOP-906 ] - a few tests need artifacts compiled against Hadoop 2.X
  • [ BIGTOP-907 ] - hadoop-mapreduce-historyserver should require hadoop-hdfs
  • [ BIGTOP-910 ] - datanode init script cannot stop the process bcause it cannot find the pid file
  • [ BIGTOP-912 ] - update HBase tests for 0.6.0
  • [ BIGTOP-914 ] - puppet needs to provision test users
  • [ BIGTOP-918 ] - Move BOM version info to the root pom file
  • [ BIGTOP-920 ] - Add HCatalog tests
  • [ BIGTOP-923] - Packages that depend on bigtop-utils needs to specify >= 0.6
  • [ BIGTOP-928 ] - Hive can't be build against non-published Hadoop artifacts
  • [ BIGTOP-929 ] - Pig component build needs to use internal resolvers
  • [ BIGTOP-930 ] - init-hdfs.sh fails on execution
  • [ BIGTOP-934 ] - Hidden cookie is being set by wiki.cloudera.org via http://blogs.apache.org/bigtop top page
  • [ BIGTOP-935 ] - Use init script templates for hive
  • [ BIGTOP-936 ] - Use init script templates for hbase
  • [ BIGTOP-937 ] - Hive does not restart on Ubuntu
  • [ BIGTOP-940 ] - fix issues with Hive package
  • [ BIGTOP-945 ] - Service lock files need to match init script name
  • [ BIGTOP-946 ] - solr-server runlevels are wrong in debian
  • [ BIGTOP-947 ] - oozie init script does not use JAVA_HOME in /etc/default/bigtop-utils
  • [ BIGTOP-948 ] - TestFileAppend and TestHDFSQuota tests need to be updated
  • [ BIGTOP-949 ] - Add Sqoop tests
  • [ BIGTOP-950 ] - race condition for output consumption in Shell code
  • [ BIGTOP-951 ] - Hive Smoke's TestJdbcDriver fails
  • [ BIGTOP-953 ] - Revert BIGTOP-835 and BIGTOP-950
  • [ BIGTOP-954 ] - metadada and user data for sqoop and hcatalog packages are wrong
  • [ BIGTOP-956 ] - fix small issues with mahout, hue and oozie tests
  • [ BIGTOP-957 ] - update deployment code for Sqoop2
  • [ BIGTOP-958 ] - update deployment code for HCatalog
  • [ BIGTOP-959 ] - get rid of hadoop specific init.d.tmpl
  • [ BIGTOP-960 ] - Remove workaround for HBASE-6263 in hbase-thrift start
  • [ BIGTOP-968 ] - oozie is now part of the core hue
  • [ BIGTOP-969 ] - a few files belong to more than one package
  • [ BIGTOP-970 ] - update package test manifest and fix small issues in package metadata for Bigtop 0.6.0
  • [ BIGTOP-971 ] - take care of the rat licensing complaints
  • [ BIGTOP-974 ] - Crunch download location is no longer /incubator
  • [ BIGTOP-975] - HBase regionserver init script does not implement "condrestart" command
  • [ BIGTOP-981 ] - Provide minimal documentation for puppet deploy config file
  • [ BIGTOP-982 ] - hcatalog needs to be given HCAT_HOME and HIVE_HOME
  • [ BIGTOP-983 ] - Oozie-client package ships its docs twice
  • [ BIGTOP-984 ] - bigtop-utils doesn't detect JAVA_HOME for Oracle JDK 7 when package by Debian's tool
  • [ BIGTOP-985 ] - Create Hive JDBC package
  • [ BIGTOP-988 ] - HCatalog does not launch daemon process in a known directory
  • [ BIGTOP-991 ] - crunch archive is incorrect
  • [ BIGTOP-994 ] - hdfs-init missed a few permissions on the sub-directories
  • [ BIGTOP-997 ] - TestCLI fails test id 5: comparator fails
  • [ BIGTOP-1001 ] - need package tests for hive-jdbc package
  • [ BIGTOP-1002 ] - Complete porting Sqoop import tests to Sqoop 2

Improvement

  • [ BIGTOP-12 ] - Add HCatalog to Bigtop
  • [ BIGTOP-547 ] - create an HDFS setup helper script
  • [ BIGTOP-712 ] - Bigtop 0.5.0 release
  • [ BIGTOP-713 ] - use newer debhelper and source format 3.0 (quilt) for Debian and Ubuntu packaging
  • [ BIGTOP-756 ] - Provide the thrift API interface file in the HBase packages
  • [ BIGTOP-763 ] - Add hdfs-journalnode to hadoop.spec
  • [ BIGTOP-817 ] - Wiki breadcrumb still shows Bigtop(incubating)
  • [ BIGTOP-821 ] - Puppet README's Usage section cuts off because of angle bracket delimiter
  • [ BIGTOP-838 ] - Both conf.empty and conf.pseudo packages don't seem to include a sample hadoop-env.sh file
  • [ BIGTOP-843 ] - It would be really cool if bigtop-detect-javahome were smarter and didn't pick a 32-bit JVM, or a 1.7 JVM set with default
  • [ BIGTOP-882 ] - Upload content of Oozie sharelib to HDFS
  • [ BIGTOP-893 ] - migrate puppet code to use init-hdfs.sh
  • [ BIGTOP-931 ] - a few improvements to secure puppet deployment code
  • [ BIGTOP-961 ] - hcatalog should be re-using hive user when running
  • [ BIGTOP-967 ] - package removal is currently missing from our package tests
  • [ BIGTOP-973 ] - Improve error diagnostics when sed'ing Hadoop version in Oozie package
  • [ BIGTOP-995 ] - JDK bug 6979329 requires a redundant kinit -R
  • [ BIGTOP-1000 ] - Remove remaining references to the Incubator

New Feature

  • [ BIGTOP-732 ] - Support running multiple HBase region servers
  • [ BIGTOP-881 ] - Add support for Oozie to be configured with SSL (HTTPS)

Task

  • [ BIGTOP-118 ] - Check to ensure our build system requirements are actually declared properly in the packaging.
  • [ BIGTOP-681 ] - Package tests needed for new DataFu package
  • [ BIGTOP-720 ] - Build environment automation
  • [ BIGTOP-760 ] - Push artifacts from non-Maven projects into local Maven cache
  • [ BIGTOP-805 ] - Add Sqoop 2 to bigtop

Test

Release Notes - Bigtop - Version 0.5.0

Sub-task

  • [ BIGTOP-39 ] - Include Apache Avro in Bigtop
  • [ BIGTOP-41 ] - Add Apache Thrift to Bigtop
  • [ BIGTOP-65 ] - Package Apache Forrest
  • [ BIGTOP-225 ] - Exit consistently in init scripts for stop on stopped service
  • [ BIGTOP-232 ] - create an example .ks file for Bigtop
  • [ BIGTOP-237 ] - Oozie init scripts are way too verbose
  • [ BIGTOP-239 ] - HBase init scripts are shut down in the wrong order
  • [ BIGTOP-240 ] - hbase initscripts should be more explicit
  • [ BIGTOP-302] - 1. Start a separate Wiki page on Installing Bigtop in VMs (VirtualBox) (make sure to record the fact that one needs >> 2GB of RAM) 2. Record the experience with 32bit OSes
  • [ BIGTOP-380 ] - Initscripts have misleading messages
  • [ BIGTOP-533 ] - BigTop Hama Integration Tests
  • [ BIGTOP-551 ] - Updates for packaging manifest
  • [ BIGTOP-612 ] - Add Crunch to Bigtop
  • [ BIGTOP-692 ] - Bump Whirr to 0.8.1
  • [ BIGTOP-695 ] - Make variables used in flume build/install scripts consistent
  • [ BIGTOP-696 ] - Make variables used in oozie build/install scripts consistent
  • [ BIGTOP-697 ] - Make variables used in zookeeper build/install scripts consistent
  • [ BIGTOP-700 ] - Make variables used in whirr build/install scripts consistent
  • [ BIGTOP-702 ] - Make variables used in datafu build/install scripts consistent
  • [ BIGTOP-703 ] - Make variables used in jsvc build/install scripts consistent
  • [ BIGTOP-704 ] - Make variables used in pig build/install scripts consistent
  • [ BIGTOP-705 ] - Make variables used in giraph build/install scripts consistent
  • [ BIGTOP-706 ] - Make variables used in hive build/install scripts consistent
  • [ BIGTOP-707 ] - Make variables used in tomcat build/install scripts consistent
  • [ BIGTOP-711 ] - update hadoop version to 2.0.2-alpha
  • [ BIGTOP-714 ] - HDFS Fuse build broken for 0.23.3
  • [ BIGTOP-733 ] - Bump Hue to 2.1.0
  • [ BIGTOP-735 ] - Upgrade to pull Solr 4.0 when it's officially released
  • [ BIGTOP-736 ] - When Solr is integrated, we'll need some tests.
  • [ BIGTOP-744 ] - add package test for Solr
  • [ BIGTOP-750 ] - Bump Zookeeper to 3.4.5
  • [ BIGTOP-751 ] - Bump HBase to 0.94.2
  • [ BIGTOP-752 ] - Bump Sqoop to 1.4.2
  • [ BIGTOP-753 ] - Bump Flume to 1.3.0
  • [ BIGTOP-765 ] - Create deployment code for Solr
  • [ BIGTOP-791 ] - Bump Oozie to 3.3.0

Bug

  • [ BIGTOP-51 ] - Layout of Bigtop repository can be optimized
  • [ BIGTOP-79 ] - Audit RPM package dependencies for programs invoked from install/uninstall scripts
  • [ BIGTOP-98 ] - Ability to force ivy/maven version inter-dependency needs to be implemented
  • [ BIGTOP-105 ] - hadoop services can not be started with the default configs in place
  • [ BIGTOP-106 ] - review and cleanup install/unistall/upgrade time dependencies
  • [ BIGTOP-107 ] - make Bigtop packaging work in kick-start environment
  • [ BIGTOP-135 ] - make jsvc use detection more robust on SLES
  • [ BIGTOP-168 ] - all of the packages that link to hadoop-core need to also link commons-configuration-1.6.jar
  • [ BIGTOP-172 ] - Hadoop init.d scripts are flaky on stop
  • [ BIGTOP-177 ] - Remove or fix versioned dependencies between packages of Bigtop
  • [ BIGTOP-187 ] - some packages ship log files
  • [ BIGTOP-200 ] - /var/lib/hadoop is set to 777 on installation it should just be group writable
  • [ BIGTOP-213 ] - We should use another group than hadoop for system groups
  • [ BIGTOP-218 ] - we need to figure out which compressors/codecs need to be pulled into Bigtop
  • [ BIGTOP-219 ] - Pig/HBase integration is not working
  • [ BIGTOP-222 ] - some man pages in /usr/share/doc/flume
  • [ BIGTOP-223 ] - install/ initscripts fixes
  • [ BIGTOP-224 ] - Some configuration files have their permissions set to 755
  • [ BIGTOP-226 ] - everything in /usr/lib/pig is set to 755. Non executable files should just be 644
  • [ BIGTOP-227 ] - whirr has all its files (except its man page) set as 755
  • [ BIGTOP-228 ] - Whirr should have a symlink to the hadoop's jars it needs
  • [ BIGTOP-229 ] - pig package install an alternative with an inconsitent name
  • [ BIGTOP-230 ] - removing packages displays the following extraneous messages
  • [ BIGTOP-256 ] - Lintian Errors/Warnings - Hadoop Package
  • [ BIGTOP-275 ] - scoping of variable in puppet code is incorrect
  • [ BIGTOP-279 ] - we shouldn't be using properties that have been deprecated in config/puppet
  • [ BIGTOP-282 ] - the licensing status of the MovieLens data files needs to be cleared up
  • [ BIGTOP-292 ] - BIgtop SRPMs can not be rebuilt
  • [ BIGTOP-307 ] - README documentation is inaccurate
  • [ BIGTOP-311 ] - hadoop does not build in off-line mode
  • [ BIGTOP-326 ] - puppet modules need to be renamed to align with renaming that happens on the package front
  • [ BIGTOP-330 ] - hadoop 0.23 pseudo conf needs to set more properties to avoid using /tmp as its datadir
  • [ BIGTOP-335 ] - sqoop-metastore service fails to start because it ends up using /usr/lib/hadoop/bin/hadoop
  • [ BIGTOP-340 ] - wrapper scripts would benefit from being explicit
  • [ BIGTOP-347 ] - we need to get rid of workaround for HDFS-1943 in hadoop init.d scripts
  • [ BIGTOP-358 ] - now that hadoop packages have been split we have to update the dependencies on the downstream packages
  • [ BIGTOP-368 ] - /usr/bin/hadoop sources /etc/default/yarn unconditionally
  • [ BIGTOP-377 ] - Clean up packages
  • [ BIGTOP-379 ] - Package testing for Bigtop 0.5.0 release
  • [ BIGTOP-385 ] - new artifacts in Hadoop 0.23 need to be properly packaged
  • [ BIGTOP-395 ] - flume-ng init.d scripts seem to be unhappy on Debian
  • [ BIGTOP-425 ] - HMaster dies from RemoteException: Server IPC version 5 cannot communicate with client version 4
  • [ BIGTOP-436 ] - flume-node stop seems to mistarget some other java process on lucid
  • [ BIGTOP-445 ] - yarn default file has the wrong name
  • [ BIGTOP-463 ] - should we reconsider /usr/lib vs. /usr/libexec decision?
  • [ BIGTOP-471 ] - hive server setup after installation fails on creating /var/lock/subsys directory
  • [ BIGTOP-483 ] - Smoke test of Hadoop fails with clitest missing testConf.xml file
  • [ BIGTOP-486 ] - separate principals from keytabs in our kerberos module
  • [ BIGTOP-489 ] - TestCLI with two test cases fails ID - 146 and 147 related to refreshServiceAcl
  • [ BIGTOP-490 ] - SmokeTest-Mahout test failures
  • [ BIGTOP-492 ] - make our launcher scripts recognize cascading defaults
  • [ BIGTOP-504 ] - provide a functionality for detecting libjvm.so in bigtop-utils
  • [ BIGTOP-511 ] - Add debian repo to bigtop puppet
  • [ BIGTOP-520 ] - HBase Debian builds run do-component-build twice
  • [ BIGTOP-525 ] - make sure yarn.dispatcher.exit-on-error is set to true
  • [ BIGTOP-531] - package metadata check error: when description is empty, the summary will append text "description" at the end of summary
  • [ BIGTOP-542 ] - Issue when installing the new zkfc
  • [ BIGTOP-545 ] - package testing manifest in trunk needs to be updated
  • [ BIGTOP-549 ] - update puppet code for the latest state of HA in Hadoop
  • [ BIGTOP-553 ] - Metadata for packages needs to be harmonized between RPM and Debain
  • [ BIGTOP-566 ] - Flume NG pkg init script should allow user to customize the location of the conf dir
  • [ BIGTOP-581 ] - Need to move iTest website to a correct location
  • [ BIGTOP-584 ] - Fix hive smoke test failures in trunk
  • [ BIGTOP-591 ] - libhdfs0-dev is missing files
  • [ BIGTOP-615 ] - Add support for Ambrose Pig job visualization
  • [ BIGTOP-624 ] - running start on a service already running should return success
  • [ BIGTOP-634 ] - Weird output when doing an init action on oozie when it is already running
  • [ BIGTOP-637 ] - Update boxgrinder appliance for the coming release
  • [ BIGTOP-644 ] - fix the runlevels for a few packages in yum/apt/zypper
  • [ BIGTOP-648 ] - hbase-thrift cannot be started properly
  • [ BIGTOP-654 ] - Set nproc for HDFS user
  • [ BIGTOP-658 ] - Move hbase dependency in hive out of the main package
  • [ BIGTOP-662 ] - cannot start hue-server
  • [ BIGTOP-663 ] - cannot start sqoop-metastore in Ubuntu precise
  • [ BIGTOP-664 ] - hive-metastore script has error when execute 'service hive-metastore status' in Ubuntu precise
  • [ BIGTOP-667 ] - start hadoop-httpfs in SLES return 3 instead of 0
  • [ BIGTOP-669 ] - Add DataFu to Bigtop distribution
  • [ BIGTOP-676 ] - Update HBase version to 0.94.1-rc0
  • [ BIGTOP-679 ] - fuse_dfs binary has moved due to HDFS-3537
  • [ BIGTOP-684 ] - copyright header in bigtop-packages/src/common/hadoop/conf.secure/core-site.xml is in wrong place
  • [ BIGTOP-686 ] - Apache Flume 1.2.0 went out so we should upgrade to it
  • [ BIGTOP-689 ] - Oozie use of /var/tmp in some CentOS versions create issues
  • [ BIGTOP-691 ] - flume gets killed too fast when the service is asked to stop
  • [ BIGTOP-694 ] - Some HBase Scripts Need To Be Excluded From Packaging
  • [ BIGTOP-699 ] - hadoop-fuse-dfs fails to find libjvm.so when JAVA_HOME happens to be a symlink
  • [ BIGTOP-701 ] - Bigtop Hadoop package does not include bash tab completion
  • [ BIGTOP-708 ] - modify Hive integration test to depends on the correct jar file
  • [ BIGTOP-721 ] - improve the package daemon status check, check twice by some delay if status doesn't match expected value
  • [ BIGTOP-725 ] - package service test TestServices should only do config once for a group of services
  • [ BIGTOP-729 ] - zookeeper initd script status doesn't work
  • [ BIGTOP-734 ] - Proposed fixes to Hive logging configuration
  • [ BIGTOP-740 ] - improve the package file content tests to ignore platform specific file names
  • [ BIGTOP-745 ] - fix incorrect javadocs in test framework
  • [ BIGTOP-754 ] - libhdfs0-dev package is empty
  • [ BIGTOP-755 ] - YARN_HOME is now HADOOP_YARN_HOME
  • [ BIGTOP-758] - [Pig] Override hadoop-core and hadoop-test artifact versions only if Hadoop < 2.x
  • [ BIGTOP-759 ] - Flume do-component-build should set hbase.version according to BOM
  • [ BIGTOP-762] - Updates for building Flume versions >= 1.2
  • [ BIGTOP-766 ] - Make it possible to override Apache URLs for the builds that require them
  • [ BIGTOP-768 ] - Bump the version of Groovy to 1.8
  • [ BIGTOP-774 ] - solr build fails on Debian
  • [ BIGTOP-775 ] - In Groovy 1.8 the behavior of a getters of static fields changed slightly
  • [ BIGTOP-776 ] - A few updates to Oozie related packaging code to accomodate Oozie's TLP status
  • [ BIGTOP-778 ] - move Solr app to under /solr in Tomcat deployment and provide more knobs
  • [ BIGTOP-780 ] - cannot install hive-server in SLES
  • [ BIGTOP-781 ] - Bigtop packages seem to have a bunch of parasitic provides
  • [ BIGTOP-782 ] - 'service hue status' still show 'failed' after hue is started in SLES
  • [ BIGTOP-783 ] - 'service hue stop' cannot stop the hue service in SLES
  • [ BIGTOP-785 ] - clean up build-time dependencies in our packages
  • [ BIGTOP-788 ] - Oozie RPM build error due to lack of usr/lib/oozie/{libserver,oozie-server,libext}
  • [ BIGTOP-789 ] - remove shadow-utils dependency from zookeeper
  • [ BIGTOP-793 ] - add crunch integration tests
  • [ BIGTOP-794 ] - puppet code around hadoop_head_node breaks on newer puppets
  • [ BIGTOP-795 ] - need to abstract away the notion of Bigtop puppet repo
  • [ BIGTOP-796 ] - hive install tries to tickle HDFS and it shouldn't
  • [ BIGTOP-797 ] - provide a way to 'rsync' files into HDFS during puppet deployment
  • [ BIGTOP-798 ] - introduce a fatjar collection of all the Bigtop integration tests and all their dependencies
  • [ BIGTOP-800 ] - BIGTOP_JSVC is lacking ARCHIVE designation
  • [ BIGTOP-801 ] - prioritize jdk over jre in bigtop-utils javadetection
  • [ BIGTOP-803 ] - somehow puppet gets confused by the hadoop fs output

Improvement

  • [ BIGTOP-12 ] - Add HCatalog to Bigtop
  • [ BIGTOP-22 ] - Unify init scripts
  • [ BIGTOP-58 ] - Add man page for Mahout
  • [ BIGTOP-60 ] - Add normal rpmbuild options for make rpm
  • [ BIGTOP-78 ] - Misc improvments
  • [ BIGTOP-117 ] - Document incompatible API changes between releases
  • [ BIGTOP-153 ] - Build should detect invalid archives in the dl directory
  • [ BIGTOP-204 ] - Add information about supported projects and criteria for support to Apache Bigtop website
  • [ BIGTOP-233 ] - Ubuntu integration: ufw application profiles
  • [ BIGTOP-245 ] - Improve Bigtop layout
  • [ BIGTOP-263 ] - we need a way to provide supervision for long running daemon applications
  • [ BIGTOP-284 ] - It would be nice to integrate Nutch crawling as one of the MR (and perhaps HBase) tests for Bigtop
  • [ BIGTOP-304 ] - document test creation and test execution steps in Bigtop
  • [ BIGTOP-351 ] - Make HBASE_HEAPSIZE manageable via puppet
  • [ BIGTOP-375 ] - Publish a table of component dependency versions
  • [ BIGTOP-410 ] - Build secure version of HBase
  • [ BIGTOP-433 ] - Make variables used in build/install scripts consistent
  • [ BIGTOP-451 ] - ooziedb.sh script should run as the unix user that runs oozie server
  • [ BIGTOP-452 ] - oozie installation should offer to run ooziedb.sh
  • [ BIGTOP-456 ] - Consider splitting homedir between mapred and hdfs users?
  • [ BIGTOP-476 ] - Improvement of BigTop iTest framework
  • [ BIGTOP-497 ] - add bash tab completion to our hadoop packages
  • [ BIGTOP-517] - Flume packaging should use the term "agent" instead of "node".
  • [ BIGTOP-536] - provide suitable title in the <title> tag of the main home page for bigtop-incubating
  • [ BIGTOP-547 ] - create an HDFS setup helper script
  • [ BIGTOP-550 ] - Add ability to have multiple zookeeper servers to puppet
  • [ BIGTOP-571 ] - we might need to guard against direct use of the upstream launcher scripts
  • [ BIGTOP-575 ] - [Pseudo-dist config] Should yarn.nodemanager.remote-app-log-dir really be set?
  • [ BIGTOP-576 ] - Minor discrepancy in naming of logs/users vs. base log directories
  • [ BIGTOP-640 ] - Sync up dependency library versions automatically
  • [ BIGTOP-670 ] - provide a link to our Jenkins server from the Bigtop's project page
  • [ BIGTOP-680 ] - Write util function to capture test input parameters
  • [ BIGTOP-688 ] - improve hue packaging via making virtual env relocatable and moving DB files into /var/lib/hue
  • [ BIGTOP-710 ] - Create a higher level orchestration deployment framework
  • [ BIGTOP-712 ] - Bigtop 0.5.0 release
  • [ BIGTOP-713 ] - use newer debhelper and source format 3.0 (quilt) for Debian and Ubuntu packaging
  • [ BIGTOP-716 ] - standardize on bash as a shell of choice
  • [ BIGTOP-718 ] - Update wiki for component requirements
  • [ BIGTOP-726 ] - make it possible to run HiveBulkScriptExecutor scripts selectively
  • [ BIGTOP-731 ] - Reorganize Hadoop tests
  • [ BIGTOP-742 ] - tighten permissions on hadoop services home directories
  • [ BIGTOP-756 ] - Provide the thrift API interface file in the HBase packages
  • [ BIGTOP-757 ] - Install ZooKeeper jars into the local Maven cache
  • [ BIGTOP-769 ] - Create a generic shell executor iTest driver
  • [ BIGTOP-772 ] - Remove all references to the incubation from the trunk of Bigtop
  • [ BIGTOP-777 ] - separate the Package Tests into service tests, file content tests, and dependency tests
  • [ BIGTOP-779 ] - create a config management utility for SolrCloud
  • [ BIGTOP-784 ] - test deployment code on all of the supported platforms (using OpenJDK7 where available)
  • [ BIGTOP-787 ] - Surface all possible options in /etc/default/flume-node
  • [ BIGTOP-792 ] - org.apache.bigtop.itest.posix.Service needs to be updated to accomodate upstart/systemd
  • [ BIGTOP-799 ] - create an itest method for unpacking all the resources into the HDFS or local filesystem
  • [ BIGTOP-802 ] - Add rollback option to DataNode service script
  • [ BIGTOP-806 ] - now that MAPREDUCE-3916 is fixed we should enable webproxy

New Feature

  • [ BIGTOP-42 ] - Start doing testpatch-like patch validation
  • [ BIGTOP-64 ] - Package Apache tools used in Bigtop builds
  • [ BIGTOP-101 ] - Need to come up with a strategy for dealing with soft dependencies between packages
  • [ BIGTOP-103 ] - add support for upstart service management facility
  • [ BIGTOP-116 ] - Add support for systemd
  • [ BIGTOP-220 ] - Add mrunit to Bigtop
  • [ BIGTOP-246 ] - Provide more verbose output at artifact build time
  • [ BIGTOP-276 ] - Standardize how clients create classpaths and java.library.paths
  • [ BIGTOP-350 ] - Some sort of system expert would be useful for manifests reviews
  • [ BIGTOP-399 ] - Provide a one-click install package for seamless deployment of Bigtop from the release/nightly repos
  • [ BIGTOP-458 ] - Create some builds/tests jobs for openJDK 6 and 7 on bigtop01 jenkins
  • [ BIGTOP-460 ] - Add support for daemontools-encore
  • [ BIGTOP-524 ] - Add tests to verify that our init scripts conform to the LSB specification
  • [ BIGTOP-594 ] - Introduce git-style bigtop command line and BIGTOP_HOME environment variable
  • [ BIGTOP-601 ] - introduce bigtop-build packages for packaging build-time dependencies
  • [ BIGTOP-635 ] - Implement a cluster-abstraction, discovery and manipulation framework for iTest
  • [ BIGTOP-685 ] - provide a way to specify the parameters expected by a test
  • [ BIGTOP-687 ] - Support YARN as a subproject of Hadoop
  • [ BIGTOP-730 ] - add tarball driver into Bigtop

Task

  • [ BIGTOP-236 ] - The user installed by hbase gets an incorrect home directory
  • [ BIGTOP-314 ] - add ZooKeeper systest to Bigtop testing regime
  • [ BIGTOP-406 ] - Enable openSuse VM on bigtop01
  • [ BIGTOP-439 ] - we need to review package test manifest for the Bigtop 0.4.0 release
  • [ BIGTOP-681 ] - Package tests needed for new DataFu package
  • [ BIGTOP-715 ] - Add Spark packaging
  • [ BIGTOP-720 ] - Build environment automation
  • [ BIGTOP-760 ] - Push artifacts from non-Maven projects into local Maven cache

Test

Wish

  • [ BIGTOP-67 ] - Resize bigtop logo in header of website

Release Notes - Bigtop - Version 0.4.0

Sub-task

  • [ BIGTOP-249 ] - Fix namenode webapp
  • [ BIGTOP-250 ] - Build contrib modules
  • [ BIGTOP-251 ] - Making sure security works with the new packages for hadoop 0.23
  • [ BIGTOP-254 ] - Write init scripts for the new yarn services
  • [ BIGTOP-268 ] - Examples do not run
  • [ BIGTOP-270 ] - Need to create versionless jars for integration point with other projects
  • [ BIGTOP-271 ] - Kill the sbin package for hadoop 0.23
  • [ BIGTOP-299 ] - Recording the fact that Java needs to be installed *before* installing Bigtop (at least on Ubuntu)
  • [ BIGTOP-300 ] - Record the information on slow mirrors
  • [ BIGTOP-387 ] - Zookeeper version needs to be updated to 3.4.3
  • [ BIGTOP-424 ] - Bump version of Oozie to 3.1.3
  • [ BIGTOP-527 ] - Integrate Hue 2.0.0 into Bigtop
  • [ BIGTOP-538 ] - update puppet code for BIGTOP-535
  • [ BIGTOP-539 ] - update puppet code for BIGTOP-534
  • [ BIGTOP-603 ] - Bump version of Pig to 0.10.0
  • [ BIGTOP-604 ] - Bump version of Hive to 0.9.0
  • [ BIGTOP-605 ] - Create How To Release document
  • [ BIGTOP-606 ] - Bump version of HBase to 0.94.0
  • [ BIGTOP-617 ] - Bump version of Oozie to 3.2.0
  • [ BIGTOP-628 ] - bump version of Mahout to 0.7.0

Bug

  • [ BIGTOP-96 ] - Mahout examples try to write to /usr/lib/mahout/examples/bin/work
  • [ BIGTOP-162 ] - downstream components need to be patched to be compatible with MR2
  • [ BIGTOP-186 ] - whirr package ships source code and needs some permissions fixes
  • [ BIGTOP-264 ] - Hadoop 23 upstream tarball put some of its hadoop jars in the same directory as some of the dependencies
  • [ BIGTOP-269 ] - in .23 HBase need to link to a much wider set of hadoop jars
  • [ BIGTOP-272 ] - small cleanup for hadoop 0.23
  • [ BIGTOP-274 ] - yarn-site.xml needs to be addedd to the Hadoop client classpath
  • [ BIGTOP-277 ] - hive package in hadoop-0.23 branch is hopelessly broken
  • [ BIGTOP-283 ] - in hadoop-0.23 the sha for Hadoop needs to be bumped to the 23.1-SNAPSHOT cut off point
  • [ BIGTOP-285 ] - Remove source jars from hadoop-0.23
  • [ BIGTOP-286 ] - hadoop-0.23: hadoop-mapreduce-examples.jar is empty
  • [ BIGTOP-288 ] - hadoop-0.23: pig contrib should have jars
  • [ BIGTOP-289 ] - hadoop-0.23: need to package native libraries for compression
  • [ BIGTOP-291 ] - __id_u RPM macro seems to be disabled in newer versions of RPMs
  • [ BIGTOP-294 ] - brp-repack-jars chokes on guice-2.0.jar because of the directories without any read permission
  • [ BIGTOP-308 ] - hadoop-0.23 branch doesn't provide .so links to hadoop native libraries
  • [ BIGTOP-309 ] - hadoop-0.23 puppet code needs to deploy history server
  • [ BIGTOP-312 ] - Puppet resource dependency syntax used in hadoop/manifests/init.pp doesn't always work
  • [ BIGTOP-316 ] - split up hadoop packages into common, hdfs, mapreduce (and yarn)
  • [ BIGTOP-325 ] - bump version of Zookeeper to 3.4.2
  • [ BIGTOP-331 ] - our init scripts fail to detect a failed history server in hadoop 0.23
  • [ BIGTOP-333 ] - heed to start properly packaging httpfs
  • [ BIGTOP-338 ] - hadoop 0.23 doc package does not contain generated documentation
  • [ BIGTOP-339 ] - zookeeper init.d script has a typo in PID file location
  • [ BIGTOP-341 ] - hadoop streaming is not available in hadoop-0.23 branch
  • [ BIGTOP-344 ] - puppet code needs to be updated to be able to deploy Hadoop 0.23 YARN in a secure fashion
  • [ BIGTOP-345 ] - container-executor.conf.dir needs to be set to /etc/hadoop/conf
  • [ BIGTOP-346 ] - container-executor needs to be owned by root:yarn and have permissions of 6050
  • [ BIGTOP-348 ] - jsvc location needs to be updated in init.d datanode script
  • [ BIGTOP-349 ] - jsvc from hadoop build fails on some of the Linux systems that Bigtop has to support
  • [ BIGTOP-355 ] - missed yarn.nodemanager.local-dirs, yarn.nodemanager.log-dirs and yarn.nodemanager.remote-app-log-dir in our puppet code
  • [ BIGTOP-357 ] - libjvm.so dependency on CentOS can only be satisfied by libgcj in some cases
  • [ BIGTOP-359 ] - documentation build is picked from the wrong place
  • [ BIGTOP-367 ] - Upgrade sequencing issue
  • [ BIGTOP-369 ] - pig package has license files in the wrong location
  • [ BIGTOP-372 ] - hive can't be started without setting HADOOP_HOME
  • [ BIGTOP-373 ] - bigtop-detect-javahome not customizable
  • [ BIGTOP-376 ] - Remove unused obsolete statement in hadoop packaging
  • [ BIGTOP-378 ] - history server security configs are missing
  • [ BIGTOP-381 ] - need to catch up with the HEAD of hadoop-common in hadoop-0.23
  • [ BIGTOP-383 ] - bigtop-detect-javahome installed at location unexpected by scripts on SLES
  • [ BIGTOP-389 ] - need to start packaging apache commons daemon jsvc
  • [ BIGTOP-392 ] - zookeeper package needs to be updated for 3.4 series
  • [ BIGTOP-393 ] - hadoop packages don't set nproc
  • [ BIGTOP-396 ] - Missing resource dependencies in puppet for secure clusters
  • [ BIGTOP-397 ] - hbase, sqoop and oozie don't use alternatives for managing configuration
  • [ BIGTOP-402 ] - manifests for package testing contain way too many duplicate data
  • [ BIGTOP-407 ] - hadoop package can't be installed on debian
  • [ BIGTOP-408 ] - HBase build needs to be worked around in hadoop-0.23 branch
  • [ BIGTOP-409 ] - sqoop build fails on SLES
  • [ BIGTOP-415 ] - need to workaround MAPREDUCE-3890
  • [ BIGTOP-416 ] - run levels for init.d scripts need to be harmonized
  • [ BIGTOP-417 ] - webapps shipped by bigtop components should be using a system-provided Tomcat
  • [ BIGTOP-418 ] - Package MAPREDUCE-2858 (MRv2 WebApp Security)
  • [ BIGTOP-420 ] - bigtop-detect-javahome installed at location unexpected by scripts on SLES
  • [ BIGTOP-421 ] - Sign every single file from the convenience artefact in the releases
  • [ BIGTOP-422 ] - need to start packaging fuse for Hadoop 0.23
  • [ BIGTOP-423 ] - hadoop package needs to be split into hadoop-client and hadoop-server packages
  • [ BIGTOP-426 ] - build failures in hadoop-0.23 branch
  • [ BIGTOP-428 ] - sqoop build needs to be forced to depend on HBase 0.92.0 artifacts
  • [ BIGTOP-429 ] - CATALINA_PID needs to be updated in hadoop-httpfs.default to correspond to the calculated value
  • [ BIGTOP-430 ] - oozie examples are missing
  • [ BIGTOP-434 ] - Potential issue in the way Bigtop test infrastructure deal with arch in packages
  • [ BIGTOP-435 ] - need to split hadoop subpackages even further for hadoop-0.23
  • [ BIGTOP-438 ] - need to bump the version of hadoop .23 to hadoop-0.23.3-SNAPSHOT
  • [ BIGTOP-443 ] - deb/oozie/oozie-client.postinst installs an alternative for a path that isn't there
  • [ BIGTOP-448 ] - Cannot install flume-ng-node pkg
  • [ BIGTOP-449 ] - hadoop package cleanup for hadoop-0.23
  • [ BIGTOP-450 ] - repackage Oozie as a true webapp
  • [ BIGTOP-453 ] - hadoop needs to depend on Zookeeper
  • [ BIGTOP-455 ] - we still need at HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec to be present in our launcher scrtips
  • [ BIGTOP-461 ] - hadoop-httpfs unconditionally removes the alternative
  • [ BIGTOP-462 ] - JSVC_HOME location can be both /usr/lib/bigtop-utils and /usr/libexec/bigtop-utils
  • [ BIGTOP-464 ] - There's currently a commented block in puppet/modules/hadoop/templates/yarn-site.xml that fails to render in some cases
  • [ BIGTOP-467 ] - [puppet] yarn historyserver currently doesn't depend on the mapreduce keytab as it should
  • [ BIGTOP-468 ] - [puppet] Fix httpfs dependencies and give it its own keytab
  • [ BIGTOP-469 ] - [puppet] web authentication principal missing from hdfs-site.xml
  • [ BIGTOP-472 ] - make deb fails when building Whirr - OutOfMemoryError: PermGen space
  • [ BIGTOP-473 ] - zookeeper man page needs to be updated
  • [ BIGTOP-474 ] - improve Oozie integration with Hadoop 0.23
  • [ BIGTOP-477 ] - launcher script for pig, hive, sqoop and mahout should not source /etc/default/hadoop anymore
  • [ BIGTOP-478 ] - perhaps we should go back to mapred as a user name for mapreduce daemons in Hadoop 0.23
  • [ BIGTOP-479 ] - init.d scripts should provide an option for initializing persistent state of the services that require it
  • [ BIGTOP-480 ] - log aggregation knob is now yarn.log-aggregation-enable and needs to be enabled
  • [ BIGTOP-481 ] - fix the way to handle repo key for Debian like system
  • [ BIGTOP-482 ] - HTTPFS puppet code needs to be updated
  • [ BIGTOP-484 ] - Fix typo error ('alredy' -> 'already')
  • [ BIGTOP-485 ] - transition package dependencies to hadoop-client package
  • [ BIGTOP-487 ] - oozie security configuration needs to be updated
  • [ BIGTOP-488 ] - TestCLI missing dependency
  • [ BIGTOP-493 ] - [puppet] core-site.xml and httpfs-site.xml end up changing every few times puppet is run
  • [ BIGTOP-496 ] - Hive ships some source code in the package.
  • [ BIGTOP-498 ] - Oozie's home dir is currently set to /var/run/oozie but should be /var/lib/oozie
  • [ BIGTOP-500 ] - Oozie hive workflows are broken because of the HADOOP_HOME being deprecated
  • [ BIGTOP-501 ] - it would help to have libext point to /var/lib/oozie in Oozie packaging
  • [ BIGTOP-502] - Flume NG init script should pass --name "agent" instead of --name "<hostname>"
  • [ BIGTOP-503 ] - remove unsupported upstream launcher scripts from packaging
  • [ BIGTOP-505 ] - Flume should exit 0 when trying to stop a non-running service
  • [ BIGTOP-506 ] - Flume should exit 0 when trying to start an already running service
  • [ BIGTOP-507 ] - sqoop deb package is empty
  • [ BIGTOP-509 ] - make all is failing because flume-1.0.0-incubating.tar.gz does not exist in APACHE_MIRROR
  • [ BIGTOP-510 ] - log4j property must be set with property in for Flume NG pkg
  • [ BIGTOP-512 ] - oozie can't be installed on debian
  • [ BIGTOP-513 ] - Pig Packaging Cleanup
  • [ BIGTOP-516 ] - it would be cleaner to get rid of mapreduce entries in limits.d/mapreduce.conf now that we consolidated the user back to mapred
  • [ BIGTOP-519 ] - /etc/default files are not marked as configs by various hadoop packages
  • [ BIGTOP-521 ] - Hadoop Pseudo-conf files have wrong permissions
  • [ BIGTOP-526 ] - it could be useful to have dfs.permissions.enabled set to true in our configs
  • [ BIGTOP-528 ] - A small patch to give better return codes which can be understood by chef and pupper among others.
  • [ BIGTOP-529 ] - [puppet] We don't currently format zookeeper nodes on first run
  • [ BIGTOP-530 ] - [puppet] We currently xst the HTTP principal multiple times, each time invalidating the previous one
  • [ BIGTOP-537 ] - provide a workaround for non-LSB compliant catalina.sh handling of a stop action
  • [ BIGTOP-540 ] - hadoop package on debian is missing JSVC_HOME specification
  • [ BIGTOP-541 ] - Build issue for hadoop RPM
  • [ BIGTOP-543 ] - fuse_dfs_wrapper.sh should not be provided
  • [ BIGTOP-544 ] - zookeeper-server init doesn't work
  • [ BIGTOP-546 ] - Hadoop rpm is missing fuse default file
  • [ BIGTOP-548 ] - we should consider moving the creation of the hive user into the hive package
  • [ BIGTOP-552 ] - Enable HDFS durable sync
  • [ BIGTOP-554 ] - rat is complaining again
  • [ BIGTOP-556 ] - hbase, yarn and hdfs users are getting incorrect home directories on Debian
  • [ BIGTOP-557 ] - /etc/oozie/conf and oozie-conf alternative should belong to oozie instead of oozie-client
  • [ BIGTOP-558 ] - debian description can't have the following set of characters ${VAR}
  • [ BIGTOP-559 ] - HttpFS fails to load Hadoop configuration after installation.
  • [ BIGTOP-560 ] - Flume-ng does not depend on ant or xml-commons anymore
  • [ BIGTOP-561 ] - HBase packaging has dangling symlinks
  • [ BIGTOP-562 ] - fix Mahout smoke test in trunk branch
  • [ BIGTOP-563 ] - Build behind proxy fails
  • [ BIGTOP-564 ] - Build failure for big-top trunk for pig, oozie, and whirr
  • [ BIGTOP-565 ] - catalina-based services should be really careful with catalina stop
  • [ BIGTOP-568 ] - Move the configuration directory for hadoop to /etc/hadoop
  • [ BIGTOP-574 ] - Add netcat and fuser as dependencies for some of the Bigtop stack.
  • [ BIGTOP-579 ] - hadoop-yarn creates log files with inconsistent ownership on Debian/Ubuntu
  • [ BIGTOP-580 ] - a wrapper for the ZooKeeper cleanup script is missing
  • [ BIGTOP-582 ] - when running Hadoop daemons pidofproc is mistakenly looking for 'su' instead of 'java'
  • [ BIGTOP-583] - "Class path contains multiple SLF4J bindings" warnings
  • [ BIGTOP-585 ] - hbase package has dangling symlinks
  • [ BIGTOP-586 ] - HBase package should provide a versionless symlink to a jar for other projects to depend upon
  • [ BIGTOP-587 ] - Hive package needs to depend on Zookeper and HBase
  • [ BIGTOP-588 ] - Oozie puppet code needs to take care of initializing the DB
  • [ BIGTOP-589 ] - TestLoadAndVerify's HBase scanner caching setting too high
  • [ BIGTOP-590 ] - hadoop man page needs to be updated
  • [ BIGTOP-592 ] - clean up content of conf.empty and conf.pseudo config files
  • [ BIGTOP-593 ] - improve the package test to handle [arch=xx] in the repo file
  • [ BIGTOP-595 ] - flume-ng agent should honor /etc/default settings
  • [ BIGTOP-596 ] - move service configuration from groovy code to package xml manifest file
  • [ BIGTOP-597 ] - Flume init scripts should not create the log and lock directories.
  • [ BIGTOP-598 ] - missing zookeeper init on debian
  • [ BIGTOP-599 ] - /var/lock/subsys may not exist on some distributions and this create some issues in Apache Hadoop
  • [ BIGTOP-600 ] - hbase init script should not source hbase-config
  • [ BIGTOP-607 ] - remove python build dependency from Hive package
  • [ BIGTOP-608 ] - remove source/build artifacts from the Pig package
  • [ BIGTOP-609 ] - add support for openjdk 1.7
  • [ BIGTOP-616 ] - Fix TestHadoopSmoke and TestHadoopExamples: tests look for streaming and examples jar in wrong directory
  • [ BIGTOP-626 ] - small misc. issues with the newly introduced Giraph package
  • [ BIGTOP-630 ] - Fix services priorities
  • [ BIGTOP-633 ] - bigtop-utils packages should depend on 'which'
  • [ BIGTOP-636 ] - a few minor tweaks to the Hue deployment code
  • [ BIGTOP-638] - move <services> block from common package_date.xml to each platform's (yum, apt, zypper) package_data.xml
  • [ BIGTOP-639] - sqoop-metastore cannot stop because "Could not load db driver class: org.hsqldb.jdbcDriver"
  • [ BIGTOP-642 ] - Suse does not have a package named which
  • [ BIGTOP-643 ] - Fix package statement in TestHDFSQuota
  • [ BIGTOP-646 ] - update metadata for Hue in the package testing manifest
  • [ BIGTOP-647 ] - service flume-node fail to start
  • [ BIGTOP-648 ] - hbase-thrift cannot be started properly
  • [ BIGTOP-649 ] - PackageTestCommon.checkRemoval() is broken
  • [ BIGTOP-650 ] - Override libexecdir for bigtop-utils on mageia

Improvement

  • [ BIGTOP-266 ] - Need to override default yarn task classpath
  • [ BIGTOP-273 ] - Need to update puppet recipies for Hadoop 0.23 deployment
  • [ BIGTOP-313 ] - Move puppet configuration out of manifest and into extlookup() CSV
  • [ BIGTOP-322 ] - update .gitignore to ignore backup files and eclipse generated files.
  • [ BIGTOP-332 ] - it would be nice to bump hadoop version to 0.23.1-SNAPSHOT across the stack
  • [ BIGTOP-366 ] - Set some minimum versions for ZK + Flume
  • [ BIGTOP-371 ] - Update bigtop repository url for the applance
  • [ BIGTOP-454 ] - update oozie man page
  • [ BIGTOP-465 ] - webhdfs isn't in the puppeted hadoop configuration
  • [ BIGTOP-466 ] - Secure zookeeper support missing from puppet
  • [ BIGTOP-470 ] - [puppet] Improve secure configuration for zk and hbase
  • [ BIGTOP-475 ] - Provides initscript for the REST HBase gateway
  • [ BIGTOP-522 ] - Minor Hive Packaging Cleanup
  • [ BIGTOP-534 ] - Need init.d script for zkfc
  • [ BIGTOP-535 ] - add httpfs config property pointing to hadoop conf dir
  • [ BIGTOP-555 ] - Bump version of HBase to 0.92.1
  • [ BIGTOP-567 ] - Make sure we can detect the JAVA_HOME for java-package built JDK package
  • [ BIGTOP-572 ] - Fix HDFS directory values in pseudo-conf hdfs-site.xml for 0.23+ NN/DN/SNN
  • [ BIGTOP-573 ] - we no longer have to override container-executor.conf.dir
  • [ BIGTOP-577 ] - [base conf] There's no hadoop-metrics2.properties present in the configs that are shipped.

New Feature

  • [ BIGTOP-248 ] - Update packages for Hadoop 0.23
  • [ BIGTOP-267 ] - Need to implement implement makefiles logic for tarball builds
  • [ BIGTOP-306 ] - Make a boxgrinder appliance
  • [ BIGTOP-405 ] - Create a bootable ISO
  • [ BIGTOP-495 ] - [puppet] Add support for HA namenodes
  • [ BIGTOP-570 ] - Build package for Apache Giraph
  • [ BIGTOP-578 ] - Integrate Hue 2.0.0 into Bigtop
  • [ BIGTOP-602 ] - Bigtop 0.4.0 release

Task

Test

  • [ BIGTOP-321 ] - Make TestLoadAndVerify parameterizable and runnable from command line
  • [ BIGTOP-444 ] - Add sqoop smoke tests
  • [ BIGTOP-569 ] - fix for TestCLI failures due to change in behavior of hadoop fs -put
  • [ BIGTOP-610 ] - Add basic hdfs fsck test and move logError helper to bigtop-test-framework
  • [ BIGTOP-611 ] - Add HBase TestImportTsv
  • [ BIGTOP-618 ] - Add HBase hbck test
  • [ BIGTOP-619 ] - Add HBase CopyTable test
  • [ BIGTOP-620 ] - Add test for dfsadmin commands
  • [ BIGTOP-622 ] - Add test for HDFS Balancer
  • [ BIGTOP-623 ] - Add test for HDFS Quota Testing
  • [ BIGTOP-627 ] - Add HBase balancer test
  • [ BIGTOP-631 ] - Add HBase Import/Export test
  • [ BIGTOP-641 ] - Add hdfs fsck options in TestFsck

Release Notes - Bigtop - Version 0.2.0

Sub-task

  • [ BIGTOP-54 ] - Bump version of oozie to 2.3.2
  • [ BIGTOP-55 ] - Bump version of sqoop to sqoop-1.3.0
  • [ BIGTOP-56 ] - Bump version of hive to hive-0.7.1
  • [ BIGTOP-80 ] - Add package dependencies to Zookeeper RPM for programs invoked from install/uninstall scripts
  • [ BIGTOP-120 ] - Bump version of pig to 0.9.1
  • [ BIGTOP-121 ] - Bump version of HBase to 0.90.4
  • [ BIGTOP-167 ] - All of Hadoop wrapper scripts and init.d scripts need to start include find-java code from bigtop-utils
  • [ BIGTOP-171 ] - All of Pig wrapper scripts and init.d scripts need to start include find-java code from bigtop-utils
  • [ BIGTOP-175 ] - All of HBase wrapper scripts and init.d scripts need to start include find-java code from bigtop-utils
  • [ BIGTOP-178 ] - All of Flume wrapper scripts and init.d scripts need to start include find-java code from bigtop-utils
  • [ BIGTOP-179 ] - All of Hive wrapper scripts and init.d scripts need to start include find-java code from bigtop-utils
  • [ BIGTOP-180 ] - All of Mahout wrapper scripts and init.d scripts need to start include find-java code from bigtop-utils
  • [ BIGTOP-182 ] - All of Oozie wrapper scripts and init.d scripts need to start include find-java code from bigtop-utils
  • [ BIGTOP-184 ] - All of Sqoop wrapper scripts and init.d scripts need to start include find-java code from bigtop-utils
  • [ BIGTOP-188 ] - All of Whirr wrapper scripts and init.d scripts need to start include find-java code from bigtop-utils
  • [ BIGTOP-190 ] - All of Zookeeper wrapper scripts and init.d scripts need to start include find-java code from bigtop-utils

Bug

  • [ BIGTOP-3 ] - gitignore files need to be removed from SVN
  • [ BIGTOP-4 ] - Remove references to Cloudera's internal infrastructure
  • [ BIGTOP-15 ] - hadoop ubuntu packages need to have libssl-dev as a build dependency
  • [ BIGTOP-20 ] - hard-coded pid location in Hadoop init.d scripts
  • [ BIGTOP-25 ] - Standardize JAVA_HOME detection
  • [ BIGTOP-27 ] - Add disclaimer text
  • [ BIGTOP-35 ] - /usr/share/doc/flume and /etc/flume are owned by flume:flume
  • [ BIGTOP-37 ] - The following config files are not marked as %config on RPM systems
  • [ BIGTOP-40 ] - iTest package names/groupIds need to be moved to org.apache rather than com.cloudera
  • [ BIGTOP-43 ] - Whirr and HBase aren't actually building from source in Debian packages
  • [ BIGTOP-45 ] - Fix build issue for Mageia and openSUSE
  • [ BIGTOP-46 ] - hadoop-native Debian package lists libz1 as Depends, libz-dev as Build-Depends
  • [ BIGTOP-47 ] - Invalid download URL for whirr
  • [ BIGTOP-48 ] - Fix pig for Mageia
  • [ BIGTOP-50 ] - Flume init script's dependency on syslog needs to be removed
  • [ BIGTOP-57 ] - Apt repo creation fails due to invalid distributions file
  • [ BIGTOP-61 ] - HBase fails to add HADOOP_CONF_DIR to its classpath
  • [ BIGTOP-62 ] - /usr/lib/hbase and everything within is owned by hbase:hbase
  • [ BIGTOP-63 ] - Hbase leaves some unnecessary directories and symlinks when being uninstalled
  • [ BIGTOP-66] - Packages need "Bigtop" identifier in version string
  • [ BIGTOP-68 ] - bigtop-tests/test-artifacts have all their code in src/test, but those aren't tests to be run in that context
  • [ BIGTOP-69 ] - Certain tests in iTest common require extraneous dependencies and should be @Ignored for now
  • [ BIGTOP-71 ] - Unit test failures in bigtop-test-framework due to multiple Ant versions on classpath
  • [ BIGTOP-72 ] - Gmaven plugin needs to be explicitly bound to compilation goals
  • [ BIGTOP-73 ] - oozie-client can't be safely removed on Debian
  • [ BIGTOP-74 ] - Add Mahout to package readiness tests
  • [ BIGTOP-75 ] - Mahout package is broken
  • [ BIGTOP-77 ] - All of our config files should be marked %config and (noreplace)
  • [ BIGTOP-81 ] - oozie-setup.sh script fails if Oozie has not been started once
  • [ BIGTOP-83 ] - Advanced checks in checkRemoval need to be moved out
  • [ BIGTOP-84 ] - Package removal fails in case alternatives are deleted or damaged
  • [ BIGTOP-85 ] - hadoop package ships native files
  • [ BIGTOP-86 ] - Misc. issues with Oozie package
  • [ BIGTOP-87 ] - The following packages don't remove alternatives: flume, hadoop-pig, hadoop-hive, hadoop-hbase
  • [ BIGTOP-88 ] - Mahout package contains actual Hadoop jar, rather than symlinking to it
  • [ BIGTOP-91 ] - Docs should be in %{_docdir}/<PACKAGE_NAME>-<PACKAGE_VERSION>
  • [ BIGTOP-92 ] - Flume's dependency on ZK needs to be the same for DEB as it is for RPM
  • [ BIGTOP-93 ] - ZK dependencies can not be satisfied on SUSE
  • [ BIGTOP-94 ] - chkconfig startup priorities need to be normalized
  • [ BIGTOP-97 ] - default files need to be normalized and moved to common
  • [ BIGTOP-99 ] - Fix Sqoop for Mageia
  • [ BIGTOP-102 ] - daemons need to be taken care of when packages are installed/upgraded/uninstalled
  • [ BIGTOP-104 ] - hadoop daemons can not be started because hadoop-config.sh resets HADOOP_HOME
  • [ BIGTOP-108 ] - oozie docs are installed at the incorrect location
  • [ BIGTOP-109 ] - Hadoop should not format the namenode on installation
  • [ BIGTOP-110 ] - DN can't be started in secure configuration on SLES
  • [ BIGTOP-113 ] - redhat-lsb is obviously not available on Mageia
  • [ BIGTOP-114 ] - Pig RPM package contains more files than it should
  • [ BIGTOP-115 ] - Pig do-component-build hardcodes dist.dir as debian/tmp
  • [ BIGTOP-119 ] - Create version-less symlinks of hadoop jars
  • [ BIGTOP-122 ] - enable hadoop-sbin package
  • [ BIGTOP-123 ] - enable hadoop-fuse package
  • [ BIGTOP-124 ] - documentation on SLES is not supposed to have package version embedded in the directory name
  • [ BIGTOP-125 ] - whir package is full of build artifacts
  • [ BIGTOP-126 ] - zookeeper rpm should require groupadd/useradd
  • [ BIGTOP-127 ] - BIGTOP-120 broke build on at least Fedora (and Mageia)
  • [ BIGTOP-128 ] - BIGTOP-90 broke hadoop on Mageia
  • [ BIGTOP-129 ] - oozie init.d scripts should transition to using oozied.sh from upstream
  • [ BIGTOP-130 ] - Mahout package has an unneeded dependency on chkconfig
  • [ BIGTOP-131 ] - RPM dependencies for flume need to be optimized
  • [ BIGTOP-132 ] - hadoop datanode on SLES returns an incorrect code on stop/start
  • [ BIGTOP-133 ] - hadoop packages should set up nofile limits automatically
  • [ BIGTOP-134 ] - hbase packages should set up nofile limits automatically
  • [ BIGTOP-136 ] - hadoop package still leaves binary files in /usr/lib/hadoop/lib
  • [ BIGTOP-137 ] - task-controller needs to be rebuilt
  • [ BIGTOP-138 ] - task-controller binary needs to be root:mapred with 4750 permissions
  • [ BIGTOP-139 ] - links to jsvc and task-controller need to be provided in order to make hard-coded values work
  • [ BIGTOP-140 ] - need to workaround HDFS-1943
  • [ BIGTOP-141 ] - hadoop now requires an explicit declaration of a HADOOP_SECURE_DN_USER in order to run a secure datanode
  • [ BIGTOP-142 ] - need to update zookeeper manifest to the state of packages
  • [ BIGTOP-143 ] - need to update hadoop manifest to the state of packages
  • [ BIGTOP-144 ] - kerberos deployment needs to be made compatible with default JRE crypto setup
  • [ BIGTOP-145 ] - make cluster.pp be able to handle secure and unsecure deployments
  • [ BIGTOP-148 ] - Hadoop fuse man page does not have the right permission and RPMs should not specify its extension
  • [ BIGTOP-149 ] - HBase now requires commons-configuration jar to be in its classpath
  • [ BIGTOP-150 ] - hadoop puppet deployment needs to install hadoop-sbin on datanodes when security is turned on
  • [ BIGTOP-151 ] - need to provide more test coverage for HBase testing
  • [ BIGTOP-152 ] - TestHadoopExamples needs refactoring into a parameterized test
  • [ BIGTOP-154 ] - all of our /usr/bin wrapper scripts and init.d scripts need to start include find-java code from bigtop-utils
  • [ BIGTOP-155 ] - need to implement a uniform way of versioning dependencies for test artifacts
  • [ BIGTOP-156 ] - We should only run autoreconf when there is a compatible autotools installed
  • [ BIGTOP-158 ] - oozie build pulls into oozie.war incorrect version of hadoop-core jar file
  • [ BIGTOP-159 ] - oozie init.d script is not executable
  • [ BIGTOP-160 ] - hadoop has deprecated HADOOP_HOME
  • [ BIGTOP-161 ] - the new layout that Pig script adopted needs to be worked around
  • [ BIGTOP-163 ] - stock hadoop doesn't support wildcards in proxy users
  • [ BIGTOP-164 ] - lzo codecs need to be disabled everywhere in Bigtop
  • [ BIGTOP-165 ] - oozie manifest includes functionality (sqoop and hive actions) that is not in oozie
  • [ BIGTOP-166 ] - docs are missing from the mahout package
  • [ BIGTOP-169 ] - Hadoop should buildrequire openssl-devel for centos builds
  • [ BIGTOP-170 ] - bigtop-utils does not look for openJDK
  • [ BIGTOP-173 ] - we still have references to hadoop-0.20 in our puppet code
  • [ BIGTOP-174 ] - Does pig really requires git to build?
  • [ BIGTOP-176 ] - Packages should set their mainteners to bigtop dev mailing list
  • [ BIGTOP-181 ] - puppeted hadoop doesn't place PID files into /var/run/hadoop
  • [ BIGTOP-183 ] - hadoop fuse manpage set to 755 instead of 644
  • [ BIGTOP-185 ] - Running 'sudo /sbin/start flume-master start' on SLES11 doesn't have $USER variable.
  • [ BIGTOP-189 ] - hadoop daemons do not run under the dedicated users (hdfs, mapred)
  • [ BIGTOP-192 ] - oozie package needs to depend on unzip
  • [ BIGTOP-193 ] - zookeeper doesn't generate any logs
  • [ BIGTOP-194 ] - source is not a sh(1) command
  • [ BIGTOP-196 ] - point Bigtop at the correct version of HBase inside
  • [ BIGTOP-197 ] - Package testing artifact needs to be refactored
  • [ BIGTOP-198 ] - hadoop-daemon.sh owned by many packages
  • [ BIGTOP-199 ] - whirr docs are installed in the incorrect location
  • [ BIGTOP-201 ] - bigtop package has an incorrect dependency on libhdfs.so
  • [ BIGTOP-202 ] - zookeeper package is missing zoo.cfg
  • [ BIGTOP-203 ] - sqoop needs to have access to hadoop .jars in its lib
  • [ BIGTOP-205 ] - zookeeper-server doesn't do nohup before launching a damon
  • [ BIGTOP-206 ] - during deployment time hdfs/mapred dir creation should be managed by hadoop module
  • [ BIGTOP-207 ] - zookeeper doesn't allow for overriding the location of the PID file
  • [ BIGTOP-208 ] - zookeeper does not depend on git or subversion
  • [ BIGTOP-211 ] - Remove packaging code that will transition hadoop user/group to hdfs/mapred
  • [ BIGTOP-212 ] - need to disable tests for features missing from Bigtop 0.2.0
  • [ BIGTOP-214 ] - sequencing in puppet module for Hadoop needs to be imporved
  • [ BIGTOP-215 ] - make site.pp a bit more friendly to how we start up cluster in Bigtop jenkins
  • [ BIGTOP-216 ] - make oozie tests more robust in finding oozie-examples.tar.gz
  • [ BIGTOP-217 ] - workaround Hive SQL parser in headers for our tests

Improvement

  • [ BIGTOP-11 ] - Add Mahout to Bigtop
  • [ BIGTOP-100 ] - Abstract component build logic from deb/rpm into common script
  • [ BIGTOP-112 ] - Remove or replace textutils by coreutils dependency
  • [ BIGTOP-195 ] - Document openSUSE 11.4 dependencies

New Feature

  • [ BIGTOP-28 ] - Need to bump upstream version dependencies for the next release of Bigtop for hive, flume and sqoop
  • [ BIGTOP-29 ] - Remove java dependency from all os types
  • [ BIGTOP-30 ] - flume-master and flume-node packages should not own /usr/lib/flume/bin/flume-daemon.sh
  • [ BIGTOP-31 ] - remove /usr/lib/whirr/cli/whirr.log from packaging
  • [ BIGTOP-32 ] - Hadoop fuse package should include a file with defaults
  • [ BIGTOP-34 ] - Add Hive server package
  • [ BIGTOP-44 ] - Add a FAQ
  • [ BIGTOP-95 ] - Puppet-based deployment infrastructure needs to be added to bigtop
  • [ BIGTOP-111 ] - Add support for Mageia in the test framework

Task

  • [ BIGTOP-59 ] - Reorganize source repo in a more self-explanatory manner

Release Notes - Bigtop - Version 0.1.0

Bug

  • [ BIGTOP-2 ] - License headers need to be added
  • [ BIGTOP-7 ] - Some components are pulling tarballs from archive.cloudera.com/tarballs rather than Apache
  • [ BIGTOP-8 ] - Debian copyright files need to be updated for ASF in the right format
  • [ BIGTOP-10 ] - Fix NOTICE file
  • [ BIGTOP-13 ] - BIGTOP-7 introduced a version mismatch for Hive
  • [ BIGTOP-18 ] - OrderedParameterized.java is not ASL-compatbile
  • [ BIGTOP-23 ] - Out of heap errors when building pig
  • [ BIGTOP-24 ] - Errors downloading tarballs from dist.apache.org via curl

Task

  • [ BIGTOP-5 ] - Remove Cloudera references in packaging and test source.
  • [ BIGTOP-6 ] - Add tarball target to top-level makefile
  • [ BIGTOP-9 ] - Set up Bigtop podling website
  • [ BIGTOP-14 ] - Add CHANGES.txt generation to Makefile