Framebuffer DT overlay is set twice

Description
I'm getting a Backtrace on boot and I'm not sure if it's related to poky/openembedded or meta-raspberrypi.
Build Configuration:
BB_VERSION           = "1.46.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "ubuntu-18.04"
TARGET_SYS           = "arm-oe-linux-gnueabi"
MACHINE              = "raspberrypi"
DISTRO               = "imst-gateway-os"
DISTRO_VERSION       = "1.0.0"
TUNE_FEATURES        = "arm armv6 vfp arm1176jzfs callconvention-hard"
TARGET_FPU           = "hard"
meta-distribution
meta-gateway         = "layer-update:0c83c87944c51441e5f7de8d72691d61e928f69e"
meta-swupdate        = "dunfell:4c82d83feacb1ad24f8e15ded5ed1a651708dd98"
meta-raspberrypi     = "dunfell:93305f7c3a9c338a3d2daef61514f61b4256f979"
meta-poky
meta-yocto-bsp       = "dunfell:4fe9a766d6357e88a024e14c732db33c9747cad5"
meta-networking
meta-webserver
meta-filesystems
meta-python
meta-oe              = "dunfell:654ad8bea49f142d20b2b96c0dd44810a6be233a"
meta                 = "dunfell:bec0a45393d968251059f5075add2cf633aecd1a"

Steps to reproduce the issue:

Build raspberrypi image with lastest dunfell branch.
Boot the image on the pi

[    1.511915] simple-framebuffer 1e513000.framebuffer: format=a8r8g8b8, mode=1824x984x32, linelength=7296
[    1.548182] Console: switching to colour frame buffer device 228x61
[    1.580552] simple-framebuffer 1e513000.framebuffer: fb0: simplefb registered!
[    1.581567] sysfs: cannot create duplicate filename '/devices/platform/1e513000.framebuffer'
[    1.581908] CPU: 0 PID: 1 Comm: swapper Not tainted 5.4.59 #1
[    1.582125] Hardware name: BCM2835
[    1.582269] Backtrace:
[    1.582419] [<c00157dc>] (dump_backtrace) from [<c0015b28>] (show_stack+0x20/0x24)
[    1.582706]  r7:00000000 r6:cb5d87a0 r5:cb51e5a0 r4:cb47b000
[    1.582942] [<c0015b08>] (show_stack) from [<c07bc1b4>] (dump_stack+0x20/0x28)
[    1.583231] [<c07bc194>] (dump_stack) from [<c0252018>] (sysfs_warn_dup+0x60/0x74)
[    1.583519] [<c0251fb8>] (sysfs_warn_dup) from [<c025216c>] (sysfs_create_dir_ns+0xec/0x10c)
[    1.583820]  r7:00000000 r6:cb51e5a0 r5:ffffffef r4:cb5e2a10
[    1.584063] [<c0252080>] (sysfs_create_dir_ns) from [<c07c14e8>] (kobject_add_internal+0x90/0x308)
[    1.584385]  r6:c1039928 r5:c1039928 r4:cb5e2a10
[    1.584586] [<c07c1458>] (kobject_add_internal) from [<c07c1c98>] (kobject_add+0x60/0xc4)
[    1.584885]  r9:c0ac282c r8:c1078000 r7:00000000 r6:c1039928 r5:cb5e2a10 r4:00000000
[    1.585203] [<c07c1c38>] (kobject_add) from [<c0523a78>] (device_add+0x100/0x61c)
[    1.585474]  r3:00000000 r2:00000000
[    1.585738]  r6:c1039928 r5:dbf62b9c r4:cb5e2a10
[    1.585964] [<c0523978>] (device_add) from [<c0648c10>] (of_device_add+0x40/0x48)
[    1.586253]  r8:c1078000 r7:00000000 r6:00000000 r5:dbf62b9c r4:cb5e2a00
[    1.586514] [<c0648bd0>] (of_device_add) from [<c0649398>] (of_platform_device_create_pdata+0xb4/0xfc)
[    1.586857] [<c06492e4>] (of_platform_device_create_pdata) from [<c0649400>] (of_platform_device_create+0x20/0x24)
[    1.587221]  r7:c09c31e0 r6:c111e8f0 r5:00000000 r4:dbf62b9c
[    1.587462] [<c06493e0>] (of_platform_device_create) from [<c0aa366c>] (simplefb_init+0x74/0x9c)
[    1.587794] [<c0aa35f8>] (simplefb_init) from [<c000ade0>] (do_one_initcall+0x50/0x1d4)
[    1.588089]  r7:00000000 r6:c1078000 r5:c0ac284c r4:c0aa35f8
[    1.598984] [<c000ad90>] (do_one_initcall) from [<c0a731c0>] (kernel_init_freeable+0x130/0x1f4)
[    1.609929]  r7:c0a6f4a0 r6:c1078000 r5:c0ac284c r4:c0aeb94c
[    1.620971] [<c0a73090>] (kernel_init_freeable) from [<c07d3af0>] (kernel_init+0x18/0xfc)
[    1.631962]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c07d3ad8
[    1.642924]  r4:00000000
[    1.653955] [<c07d3ad8>] (kernel_init) from [<c00090ac>] (ret_from_fork+0x14/0x28)
[    1.664988] Exception stack(0xcb4cbfb0 to 0xcb4cbff8)
[    1.675936] bfa0:                                     00000000 00000000 00000000 00000000
[    1.687241] bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    1.698351] bfe0: 00000000 00000000 00000000 00000000 00000013 00000000
[    1.709317]  r5:c07d3ad8 r4:00000000
[    1.720413] kobject_add_internal failed for 1e513000.framebuffer with -EEXIST, don't try to register things with the same name in the same directory.

Can you post Build Configuration from bitbake ? secondly is it using dunfell branch for all repos ?

Can you post Build Configuration from bitbake ? secondly is it using dunfell branch for all repos ?

Im using Dunfell for all branches. What exactly do you mean with build configuration? I'm using a custom image with own layer. There is nearly nothing in my local.conf

Can you post Build Configuration from bitbake ? secondly is it using dunfell branch for all repos ?

Im using Dunfell for all branches. What exactly do you mean with build configuration? I'm using a custom image with own layer. There is nearly nothing in my local.conf

since the description does not have complete info to reproduce it, what bitbake prints at the beginning of build is a good information to stick in such reports always .

Can you post Build Configuration from bitbake ? secondly is it using dunfell branch for all repos ?

Im using Dunfell for all branches. What exactly do you mean with build configuration? I'm using a custom image with own layer. There is nearly nothing in my local.conf

since the description does not have complete info to reproduce it, what bitbake prints at the beginning of build is a good information to stick in such reports always .

I edited the first post.

@kraj do you have any idea to fix it. I have been looking for a solution for hours.

it boots fine here on rpi4/32bit as well as 64bit so I dont know how can I help

it boots fine here on rpi4/32bit as well as 64bit so I dont know how can I help

Have somebody test it with raspberry pi 1?
Do you have some experience in debugging issues like this?

Does Raspbian boot on your board? Can you compare the kernel versions? Are you using uboot? If yes, can you test without it?

Does Raspbian boot on your board? Can you compare the kernel versions? Are you using uboot? If yes, can you test without it?

It's building in this moment a version without U-Boot and any customization. I need U-Boot for sw-update but for testing purpose I removed it also. The kernel version is 5.4.59.
I will give you a update when it's finished.

Something new about the issue?

Sadly there is nobody now working on RPi1. I roughly know what the issue is but I recommend you bring it up to the uboot guys as they will be able to fix it in upstream and we can just update and take advantage of the fix.

Sadly there is nobody now working on RPi1. I roughly know what the issue is but I recommend you bring it up to the uboot guys as they will be able to fix it in upstream and we can just update and take advantage of the fix.

What I get as an answer from u-boot IRC.
what does RPI_USE_U_BOOT = "1" relate to? I would have expected kernel=u-boot.bin in config.txt. Which version of U-Boot are you using? Which U-Boot commands do you use for booting? You can find the address of the RPi maintainer for U-Boot in file /board/raspberrypi/rpi/MAINTAINERS. Maybe you want to send a mail with all details to him and the U-Boot list.
sysfs: cannot create duplicate filename '/devices/platform/1e513000.framebuffer'
this is your problem
dump the DT passed to the kernel and check for duplicate nodes for that framebuffer device

Sadly there is nobody now working on RPi1. I roughly know what the issue is but I recommend you bring it up to the uboot guys as they will be able to fix it in upstream and we can just update and take advantage of the fix.

I talked a lot with the U-Boot guys and the sad everything is working. The recommended me to find the issue yourself. Thereupon I have also tried to fix it, but my knowledge Bout the Linux kernel is to bad. You have mentioned that you know about the issue, can you give me some hints? Which recipes I need to modify with bitbake. I looked in U-Boot and there is nothing about a framebuffer definition. It's also possible that the framebuffer is defined in the prebuild DT files from raspberry pi firmware.

@piechade any progress on this?


lib.types does not reveal all types

What happened
I'm currently working on adding documentation to lib/types.nix in nixpkgs.
Not all lib.types are shown on noogle.
I assume this is due to limitations of nixdoc?
That being said finding types for use with nixos modules is quite valued.
One other question is should I document the types that are not explicit functions In the interim assuming this may get solved in the future.
Steps to Reproduce
Steps to reproduce the behavior:

Go to noogle.dev
Compare with here
See the type not present

Expected behavior
Expected behavior is all types show up even if not functions.
Desktop (please complete the following information):

OS: nixos
Browser Firefox
Version: 1.0.0-a.39

Hey thanks for opening this issue.
Noogle was built as a search for functions. Those functions that are defined by lib.types should be displayed.
But i am also Open to extend the scope carefully and also display a selection of important attributes auch as lib.types.
If you need any help with the documentation just mention me in the nixpkgs PR.

Noogle was built as a search for functions. Those functions that are defined by lib.types should be displayed.

I'm glad you're willing to expand in specific cases such as this! types is something I currently have to frequent when working on nixos modules.

If you need any help with the documentation just mention me in the nixpkgs PR.

i'll be certain to ping when the time comes. there is a lot I can deduce as of right now just by grepping the repo and finding examples uses as i go.
The question remains if I should code comment the non function types in preparation for such a feature?

Documentation for most types is here:
https://nixos.org/manual/nixos/stable/#sec-option-types
No need to rewrite. We just need to factor it in the long term

Documentation for most types is here: https://nixos.org/manual/nixos/stable/#sec-option-types No need to rewrite. We just need to factor it in the long term

understood! I'll port them over to be as code comments, or are you able to grab them from their directly for noogle?


Card's API docs: move screenshots above the sample code

From a recent study, we found screenshots helped participants immediately determine if this is the right UI widget to use. Next, they looked for code samples to learn how to use a widget.
In the current Card widget API doc, participants needed to scroll back and forth a couple of times to notice the screenshot which is displayed below the code.
Proposed order:

(introduction)
screenshot
sample code

cc: @gspencergoog as you recently updated the sample code UI on this page.

Seems this is fixed already. Can you please check?

Yes, this is fixed.


SocketException: Permission denied +Questions

Issue Title
Socket Exception
General
Hi,
I'm new at the programing for raspberry at the moment i'm porting code from netduino to raspberry and i ran on problem with socketException: Permission denined. What can be the reason for this? I have try to change port to other and stil get the same problem.
Just for info I'm running on dot net core 2.2.3.
I have also question if is there any option to debug on raspberry and what is procedure to setup app that will run on raspberry each time when machine restart.
Thanks.

Can you provide more details?  What operating system is your raspberry pi running? Can we see a code example of what you're trying to do?

3 weeks no response, closing. Feel free to reopen if there is more info available. Thanks!


Added call expression types to typescript

Checklist

[x] I have added tests
[-] I have updated the docs and cheatsheet
[-] I have not broken the cheatsheet

please do decl side before mergning tho

what happens with
foo<Bar<Baz, Bongo>>()
Foo<Bar>
Bar<Baz, Bongo>
I think probably ok to support more of these cases

It would also be good to have a test to formalize what happens with multiple type args

@pokey Ready for second look

[ ] add test for const foo: Record<string, string> = {} on hte first string

@pokey updated


DevOps Pipeline Fails With Error: The process '/usr/bin/bash' failed with exit code 1

Describe the bug
My DevOps pipeline just started failing today (last successful build was last night). The project is a react front end with javascript function endpoints. The build fails after uploading to the app service. From what I can tell, it times out when polling to see if the upload finished (see screenshot). The code still gets deployed though.
To Reproduce
Steps to reproduce the behavior:

Check in a code change.
Push to DevOps repo
Kick off build pipeline

Azure Pipelines YAML File
trigger:
  - master

pool:
  vmImage: ubuntu-latest

steps:
  - checkout: self
    submodules: true
  - task: AzureStaticWebApp@0
    inputs:
      app_location: '/'
      output_location: 'build'
      api_location: 'api'
      verbose: true
      azure_static_web_apps_api_token: '$(deployment_token)'

Expected behavior
Build succeeds without errors
Screenshots

We are working on fixing an issue related to repositories that have a hostname of '.visualstudio.com', does your repo fall into that category?

it does indeed

Yup we are actively working on fixing this, sorry for the inconvenience!

We just finished rolling out a mitigation, you should be able to retry successfully now.

@miwebst We are setting the same issue with Static Web Apps + GitHub connection and we are seeing the above message.

Affected also, same error as of this posting.  Glad to hear this issue is already open, will stand by to test when the next fix rolls out!

@miwebst Very similar to the above, yes.  Specifics are:

jQuery project in GitHub Repo.  Includes three azure-pipelines.yml files to three different Azure Static Web Apps (production, staging, development) based on the trigger/branch
Task Name: AzureStaticWebApp@0
NPM based, uses gulp, to do some transformation of the project

We have just applied another hotfix, please retry and let us know if you are unblocked!

Just tested, successfully deployed to the static web app!
Thanks for resolving the issue 👍

Same.  All is good now.  Thanks!

Still seeing this. Is it back? My Static Website succeeds on a PR test, but fails (upload timed out) on release builds. Any ideas? Same .yml for both.

experiencing the issue now.
##[error]Error: The process '/usr/bin/bash' failed with exit code 1
Upload Timed Out. Unsure if deployment was successful or not.

I am getting the same error trying to deploy static web app from azure repos


Cranelift: JIT relocations depend on system allocator behaviour

Hey,
I'm seeing crashes during finalize_definitions calls related to x86_64 call relocations:
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: TryFromIntError(())', cranelift/jit/src/compiled_blob.rs:55:80

Cranelift emits 32-bit relocations for calls on x86_64, and thus can "only" address in the relative ±2GB range. Code memory is allocated with the normal system allocator, which might place different allocations in distant parts of the address space.
I'm seeing irregular crashes in a heavily multithreaded program, but the problem can be reproduced with this abridged jit-minimal.rs example:
use cranelift::prelude::*;
use cranelift_codegen::settings;
use cranelift_jit::{JITBuilder, JITModule};
use cranelift_module::{default_libcall_names, Linkage, Module};

fn main() {
    let isa_builder = cranelift_native::builder().unwrap();
    let isa = isa_builder
        .finish(settings::Flags::new(settings::builder()))
        .unwrap();

    let mut m = JITModule::new(JITBuilder::with_isa(isa, default_libcall_names()));

    let mut ctx = m.make_context();
    let mut func_ctx = FunctionBuilderContext::new();

    let func_a = m
        .declare_function("a", Linkage::Local, &m.make_signature())
        .unwrap();
    let func_b = m
        .declare_function("b", Linkage::Local, &m.make_signature())
        .unwrap();

    // Define a dummy function `func_a`
    ctx.func.name = ExternalName::user(0, func_a.as_u32());
    {
        let mut bcx: FunctionBuilder = FunctionBuilder::new(&mut ctx.func, &mut func_ctx);
        let block = bcx.create_block();

        bcx.switch_to_block(block);
        bcx.ins().return_(&[]);
        bcx.seal_all_blocks();
        bcx.finalize();
    }
    m.define_function(func_a, &mut ctx).unwrap();
    m.clear_context(&mut ctx);

    // Allocate a bunch (~4GB) to stretch address space
    let mut allocations: Vec<Vec<u8>> = Vec::new();
    for _ in 0..999999 {
        allocations.push(Vec::with_capacity(4096));
    }

    // Define `func_b` in a new allocation and reference `func_a`
    ctx.func.name = ExternalName::user(0, func_b.as_u32());
    {
        let mut bcx: FunctionBuilder = FunctionBuilder::new(&mut ctx.func, &mut func_ctx);
        let block = bcx.create_block();

        bcx.switch_to_block(block);
        let local_func = m.declare_func_in_func(func_a, &mut bcx.func);
        // Emit a call with a relocation for func_a
        bcx.ins().call(local_func, &[]);
        // Make sure that this function's body is larger than page_size and will require a new allocation.
        for _ in 0..1024 {
            bcx.ins().call(local_func, &[]);
        }
        bcx.ins().return_(&[]);

        bcx.seal_all_blocks();
        bcx.finalize();
    }

    m.define_function(func_b, &mut ctx).unwrap();
    m.clear_context(&mut ctx);

    // Perform linking
    m.finalize_definitions();
}

It might be possible to trigger this from small-ish WebAssembly modules with glibc's mmap threshold that places >128kb allocations outside of the heap, though I haven't had any luck reproducing that because glibc's dynamic threshold scaling raises this limit before code is emitted.
Possible approaches:

Determine the total size of the finalized code page before allocating; allocate one large chunk. It seems like an implementation for this should be doable, though I'm not sure if this is by design. (This would be incompatible with features like hot function replacement.)

Don't allocate on the heap. Cranelift's selinux-fix features uses mmap allocations. The underlying issue still persists, though as mmap allocations are separate from the heap, they're mostly sequential and would need >2GB of generated machine code to cause problems.

(Change relocation style? There's no 64-bit relative jump in x86_64 and blowing up code size for this seems like a bad idea.)

Aarch64 runs into a related issue with 26-bit relative jumps: https://github.com/bytecodealliance/wasmtime/issues/3277
I'm not sure veneers are applicable for x86_64, but they seems like an interesting and more general approach to relative jump range limits.

Don't allocate on the heap. Cranelift's selinux-fix features uses mmap allocations. The underlying issue still persists, though as mmap allocations are separate from the heap, they're mostly sequential and would need >2GB of generated machine code to cause problems.

I think this is the best fix. Possibly in combination with reserving the full 2GB as PROT_NONE. Allowing the GOT to be split between each such 2GB chunk should also allow more code to be used when PIC is enabled.

That is not the problem here. The problem is that a function and the GOT or PLT it accesses may end up more than 2GB from each other due to memory fragmentation. All calls already go through the GOT and PLT anyway so for as long as those are within 2GB it doesn't matter where the function is, independent of the colocated flag.


Simplify Parsec Code

Coverage decreased (-1.5%) to 94.643% when pulling 2662a25444277357661abd004d48af3909596177 on enhance_tests into 363630bb98f4f71d48a0400e4a449564a0aa9668 on main.


test: re-enable openfhe end to end tests after #727 is fixed upstream

Here is the upstream WIP https://github.com/llvm/llvm-project/pull/94838
So far I've reproduced the error but don't know how to fix it.

Can reenable once https://github.com/llvm/llvm-project/pull/96242 is integrated

The upstream fix is merged, but I'm still getting segfaults.


Added golang base server

Saw this project on my github and decided I'd add something to it. :) hope you don't mind.
Thinking about adding elm-lang as the front end. Also not sure what css framework would be good or if it is even needed.

I think I might add http://bulma.io/ as the css UI. It uses CSS Flexbox. I think it would speed up the UI creation a lot.

Sure thing! One of many projects that need some love. What exactly does elm-lang use? Is it part of golang? And I like Bulma and how fluid it is. Might want to change the colour scheme though.

So elm is a functional language that compiles to javascript. It's really handy.
The whole thing is built around a loop "data updates -> message is triggered -> message is handled -> data updates"
It also holds the html structure in the elm file. So a div would be Html.div [ class "hi" ] [ text "Hey" ] if you want a div within a div it would be Html.div [ class "first div" ] [ Html.div [ class "second div" ] [ text "hey" ] ]
I've used Vue.js and Angular and have only realy felt at ease coding with elm. We of course can use Vue.js, Angular, React, or plain javascript if you would like. Other discussion should be done on issue #3 .


revision of coordinateSpace according to #40

Please review and double check.

Also, Wiki needs to be updated. It's not reflecting these changes 100%.

I'll update the Wiki after the PR is merged. Thanks for the reminder


Problems with overriders in api with release 8.1.0

Before opening, please confirm:

[X] I have installed the latest version of the Amplify CLI (see above), and confirmed that the issue still persists.
[X] I have searched for duplicate or closed issues.
[X] I have read the guide for submitting bug reports.
[X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
[X] I have removed any sensitive information from my code snippets and submission.

How did you install the Amplify CLI?
npm
If applicable, what version of Node.js are you using?
v16.14.0
Amplify CLI Version
8.1.0
What operating system are you using?
Ubuntu
Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.
When I try to push
Amplify Categories
api
Amplify Commands
push
Describe the bug
Today I updated the amplify cli from version 8.0.3 to version 8.1.0.
Since then my project is giving errors when doing push, because it is not ordering well the pipeline fuctions of my api. I use custom vtl as the documentation says in this link https://docs.amplify.aws/cli/graphql/custom-business-logic/#extend-amplify-generated-resolvers, it worked great until I updated, the new version does not order correctly the sequence of the slot preDataLoad.
I have already downgraded the CLI to 8.0.3 and it is working great again.
Thank you.
Expected behavior
in my overrider I replace the dataSourceName
Reproduction steps
1.- amplify push
GraphQL schema(s)

# Put schemas below this line

Log output

# Put your logs below this line

Additional information
No response

Hey @eliecer2000 , following up on this to see if you are experiencing any issues.

Closing the issue due to inactivity. Please do reach out to us again and we would be happy in reopening the issue.

@eliecer2000, et. al.
I encountered this issue too and finally figured out what is happening.
FYI, I am using an ancient version (on this day of the lord, Oct 15 2024) 12.3.0. It seems that amplify is hashing the VTL templates and then only associating templates that are different to the pipelines. If a template is "the same" as another, you may get functions in a pipeline with a different name than you may expect because of this behavior.
The solution is to ensure that each template does in-fact have different contents.


Request SSIM results

Hi, could you please provide the ssim results for table 2? Or could you share the swinir-metasr  and swinir-rdn pretrained models?  Thanks!

Hi, @543350391. We have uploaded SwinIR-MetaSR and SwinIR-LIIF pertained models.


Problems Making Certificate Request - string is too long, it needs to be no more than 64 bytes long

In raising this issue, I confirm the following:
{please fill the checkboxes, e.g: [X]}

[X] I have read and understood the contributors guide.
[X] The issue I am reporting can be replicated.
[X] The issue I am reporting can be is directly related to the pivpn installer script.
[X] The issue I am reporting isn't a duplicate (see FAQs, closed issues, and open issues).

Issue
Install script fails because EC private key filename is too long
Have you searched for similar issues and solutions?
(yes/no / which issues?)

Yes
Console output of      curl -L install.pivpn.io | bash
::: Installing scripts to /opt/pivpn... done.
set_var EASYRSA_ALGO       ec
set_var EASYRSA_CURVE      prime256v1
::: Building CA...
read EC key
writing EC key
Can't load /etc/openvpn/easy-rsa/pki/.rnd into RNG
140632924271744:error:2406F079:random number generator:RAND_load_file:Cannot open file:../crypto/rand/randfile.c:98:Filename=/etc/openvpn/easy-rsa/pki/.rnd

::: CA Complete.

Note: using Easy-RSA configuration from: ./vars

Using SSL: openssl OpenSSL 1.1.1c  28 May 2019
Generating an EC private key
writing new private key to '/etc/openvpn/easy-rsa/pki/private/pihole-northamerica-northeast1-1_5a93a934-9f11-4ff7-80f4-dc8f61d60156.key.S9pvXM64Sz'
-----
string is too long, it needs to be no more than 64 bytes long
problems making Certificate Request

Easy-RSA error:

Failed to generate request
./easyrsa: 319: set: Illegal option -o echo

Console output of      pivpn add or pivpn add nopass
  Output Here

Console output of      pivpn debug
n/a

Have you taken any steps towards solving your issue?
No

fix for this pushed into test branch, please checkout and test it. thank you.

this has been merged today into Master branch.
Closing


Bugfix/#4 optimize information cards in mobile view

@thatfreakcoder, I added some CSS for a clean result in all resolutions. It should now look OK, instead of all messed up :-).
I also added a small fix in the README (broken link to CONTRIBUTING), and there was still an unresolved conflict in the footer.html file on master.
I have also noticed that there's some other weird stuff going on with the size of the cards when rescaling the window and when hovering over the cards, but I left that alone for now.

Hey @tmttn I've reviewed the commits. Thanks for updating :)


[MINOR][PERF] String new line error

While running performance tests on AWS, I noticed that a new line was being added to the end of the string.
This caused weird file names (m-svm.multinomial.dense.10k_100?.0.json) and errors during the perf test execution.
By stripping the string this problem was resolved and could run my performance test workloads.

thanks for the review @gweidner

Hi @krishnakalyan3 , can the commit message be a little more explanatory, when we look at the git log.
Thanks.

@j143 yes. Thanks for the feedback.


Add MultiFileChooser to the README widgets table

It took me a while to figure out that this existed, and having it in this table would have helped!

Awesome. Thanks! 👍


Update base_preprocessing.py

Potential bug - please verify

Not a bug ;)
We only train on train+val jointly, for the benchmark setting.


Make default implementation tcnn

Hi, can you make all the implementation: [tcnn, torch] arguments default to tcnn (e.g. HashMLPDensityField and nerfacto are not)? Upgraded to the latest nerfstudio and spent quite a while trying to figure out why my code was 4x slower.

Yes indeed. Although it seems like the default in nerfacto is already tcnn, do you mind linking where you are referring to?

Maybe the instant-ngp implementation? https://github.com/nerfstudio-project/nerfstudio/blob/6fc588bda3d14d01e824947d67e2e1143a30fa58/nerfstudio/models/instant_ngp.py#L111-L118

https://github.com/nerfstudio-project/nerfstudio/blob/d6bb8bd8e5d278634b5fa73c18ae8bcc0daad986/nerfstudio/fields/density_fields.py#L34
This one-- might be a tricky one for others who are also upgrading


Add html for MiniSelectKind

html!{} is missing for MiniSelectKind::MoreActionNoImage. Add the MoreAction html!{}

@marioCluml this is a public repository, so you should not attach screenshots that would expose information about non-public repositories. Your screenshot contains such info. Please crop that info out of the screenshot.

@marioCluml this is a public repository, so you should not attach screenshots that would expose information about non-public repositories. Your screenshot contains such info. Please crop that info out of the screenshot.

My apologies, I will crop it right away

To make it work, it seems like we need part like this as well: https://github.com/aicers/frontary/blob/main/src/select/mini.rs#L269-L290


Skip tests that require daemonset

I think I have covered all of the test cases that require using the debug daemonset pods to retrieve information.
Changes:

Sets a variable env.DaemonsetFailedToSpawn if there's a problem with the daemonset coming up.
Skips tests that require the daemonset to complete.
Removes leftover function DeployPartnerTestDaemonset which is no longer needed since we are using the #621 change for the privileged daemonset library.  Delete the internal/daemonset package and files.

@ramperher Thanks!  Fixed the additional tests in https://github.com/test-network-function/cnf-certification-test/commit/b3375e127e8b7ab8a600c81b3613b9447cbfceaa


New version: GEMPIC v0.2.3

Registering package: GEMPIC
Repository: https://github.com/JuliaVlasov/GEMPIC.jl
Created by: @pnavaro
Version: v0.2.3
Commit: be22d5f6937a71891062f2b0ab964a0b14291c48
Reviewed by: @pnavaro
Reference: https://github.com/JuliaVlasov/GEMPIC.jl/issues/8#issuecomment-792997793

@pnavaro :point_up:
[noblock]


Add management console deprecation banner

Purpose

Add a notice banner to indicate that management console is being deprecated and only a limited set of features will be provided through this.

This is by default disabled and enabled through the following deployment config.
[management_console_deprecation_banner]
enable = true

Issue
https://github.com/wso2/product-is/issues/18950

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/7843098262

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/7843098262
Status: success


Error when running imported/restored model that uses feedable iterator (tf.contrib.data)

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10
TensorFlow installed from (source or binary): Binary
TensorFlow version (use command below): Build #242 (Jul 17, 2017 2:25:00 AM)
Python version: Python 3.6.2
Bazel version (if compiling from source): na
CUDA/cuDNN version: 8.0/6.0
GPU model and memory: 670 gtx 2gb
Exact command to reproduce: na

Describe the problem
I can't restore and run checkpoints of models that use feedable iterators, but I can restore and run checkpoints of models that directly use make_one_shot_iterator(). Below is code for the feedable iterator version, and below that code for the make_one_shot_iterator() version:
import tensorflow as tf
from tensorflow.contrib.data import Dataset

BATCH_SIZE = 4
ITERATION_COUNT = 2
dataset = Dataset.from_tensor_slices(tf.constant([[0, 0],
                                                  [0, 1],
                                                  [1, 0],
                                                  [1, 1]], dtype=tf.float32))
batched_dataset = dataset.batch(BATCH_SIZE)
iterator_handle_placeholder = tf.placeholder(tf.string, shape=[])
tf.add_to_collection('placeholders', iterator_handle_placeholder)
iterator = tf.contrib.data.Iterator.from_string_handle(iterator_handle_placeholder, batched_dataset.output_types, batched_dataset.output_shapes)

# create some graph
inputs = iterator.get_next()
sum_placeholder = tf.placeholder_with_default(tf.reduce_sum(inputs), shape=[])
tf.add_to_collection('placeholders', sum_placeholder)
sum_variable = tf.get_variable('sum', initializer=tf.zeros(shape=[]))
assign_sum = tf.assign_add(sum_variable, sum_placeholder)
tf.add_to_collection('assigns', assign_sum)
saver = tf.train.Saver()

# run graph and save it at the end
with tf.Session() as session:
    session.run(tf.global_variables_initializer())
    batched_dataset_iterator = batched_dataset.make_one_shot_iterator()
    batched_dataset_iterator_handle = session.run(batched_dataset_iterator.string_handle())
    tf.add_to_collection('handles', batched_dataset_iterator_handle)
    inputs_feed_dict = {iterator_handle_placeholder: batched_dataset_iterator_handle}
    for i in range(ITERATION_COUNT):
        inputs_sum = session.run(assign_sum, feed_dict=inputs_feed_dict)
        inputs_feed_dict[sum_placeholder] = inputs_sum
        print(inputs_sum)
    saver.save(session, 'checkpoints/haha')

# restore saved graph and run it
with tf.Session() as session:
    saver = tf.train.import_meta_graph('checkpoints/haha.meta')
    saver.restore(session, 'checkpoints/haha')
    assign_sum = tf.get_collection('assigns')[0]
    iterator_handle_placeholder = tf.get_collection('placeholders')[0]
    batched_dataset_iterator_handle = tf.get_collection('handles')[0]
    sum_placeholder = tf.get_collection('placeholders')[1]
    inputs_feed_dict = {iterator_handle_placeholder: batched_dataset_iterator_handle}
    for i in range(ITERATION_COUNT):
        inputs_sum = session.run(assign_sum, feed_dict=inputs_feed_dict)
        print(inputs_sum)
        inputs_feed_dict[sum_placeholder] = inputs_sum

gives me
C:\Users\Jonathan\Miniconda3\envs\ai\python.exe "C:/Software Projects/ai/tensorflow/LayerTests.py"
2017-07-21 20:18:54.013753: W C:\tf_jenkins\home\workspace\nightly-win\M\windows-gpu\PY\36\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-07-21 20:18:54.257354: I C:\tf_jenkins\home\workspace\nightly-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:955] Found device 0 with properties:
name: GeForce GTX 670
major: 3 minor: 0 memoryClockRate (GHz) 0.98
pciBusID 0000:02:00.0
Total memory: 2.00GiB
Free memory: 1.64GiB
2017-07-21 20:18:54.257634: I C:\tf_jenkins\home\workspace\nightly-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:976] DMA: 0
2017-07-21 20:18:54.257773: I C:\tf_jenkins\home\workspace\nightly-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:986] 0:   Y
2017-07-21 20:18:54.257935: I C:\tf_jenkins\home\workspace\nightly-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 670, pci bus id: 0000:02:00.0)
4.0
8.0
2017-07-21 20:18:54.705095: I C:\tf_jenkins\home\workspace\nightly-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 670, pci bus id: 0000:02:00.0)
2017-07-21 20:18:54.774833: W C:\tf_jenkins\home\workspace\nightly-win\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Not found: Container localhost does not exist.
Traceback (most recent call last):
  File "C:\Users\Jonathan\Miniconda3\envs\ai\lib\site-packages\tensorflow\python\client\session.py", line 1327, in _do_call
    return fn(*args)
  File "C:\Users\Jonathan\Miniconda3\envs\ai\lib\site-packages\tensorflow\python\client\session.py", line 1306, in _run_fn
    status, run_metadata)
  File "C:\Users\Jonathan\Miniconda3\envs\ai\lib\contextlib.py", line 88, in __exit__
    next(self.gen)
  File "C:\Users\Jonathan\Miniconda3\envs\ai\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 466, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.NotFoundError: Container localhost does not exist.
	 [[Node: IteratorFromStringHandle = IteratorFromStringHandle[_device="/job:localhost/replica:0/task:0/cpu:0"](_arg_Placeholder_0_0)]]
	 [[Node: PlaceholderWithDefault/_7 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_7_PlaceholderWithDefault", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Software Projects/ai/tensorflow/LayerTests.py", line 47, in <module>
    inputs_sum = session.run(assign_sum, feed_dict=inputs_feed_dict)
  File "C:\Users\Jonathan\Miniconda3\envs\ai\lib\site-packages\tensorflow\python\client\session.py", line 895, in run
    run_metadata_ptr)
  File "C:\Users\Jonathan\Miniconda3\envs\ai\lib\site-packages\tensorflow\python\client\session.py", line 1124, in _run
    feed_dict_tensor, options, run_metadata)
  File "C:\Users\Jonathan\Miniconda3\envs\ai\lib\site-packages\tensorflow\python\client\session.py", line 1321, in _do_run
    options, run_metadata)
  File "C:\Users\Jonathan\Miniconda3\envs\ai\lib\site-packages\tensorflow\python\client\session.py", line 1340, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Container localhost does not exist.
	 [[Node: IteratorFromStringHandle = IteratorFromStringHandle[_device="/job:localhost/replica:0/task:0/cpu:0"](_arg_Placeholder_0_0)]]
	 [[Node: PlaceholderWithDefault/_7 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_7_PlaceholderWithDefault", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

Caused by op 'IteratorFromStringHandle', defined at:
  File "C:/Software Projects/ai/tensorflow/LayerTests.py", line 13, in <module>
    iterator = tf.contrib.data.Iterator.from_string_handle(iterator_handle_placeholder, batched_dataset.output_types, batched_dataset.output_shapes)
  File "C:\Users\Jonathan\Miniconda3\envs\ai\lib\site-packages\tensorflow\contrib\data\python\ops\dataset_ops.py", line 238, in from_string_handle
    string_handle)
  File "C:\Users\Jonathan\Miniconda3\envs\ai\lib\site-packages\tensorflow\python\ops\gen_dataset_ops.py", line 362, in iterator_from_string_handle
    string_handle=string_handle, name=name)
  File "C:\Users\Jonathan\Miniconda3\envs\ai\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 767, in apply_op
    op_def=op_def)
  File "C:\Users\Jonathan\Miniconda3\envs\ai\lib\site-packages\tensorflow\python\framework\ops.py", line 2628, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "C:\Users\Jonathan\Miniconda3\envs\ai\lib\site-packages\tensorflow\python\framework\ops.py", line 1204, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

NotFoundError (see above for traceback): Container localhost does not exist.
	 [[Node: IteratorFromStringHandle = IteratorFromStringHandle[_device="/job:localhost/replica:0/task:0/cpu:0"](_arg_Placeholder_0_0)]]
	 [[Node: PlaceholderWithDefault/_7 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_7_PlaceholderWithDefault", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

Process finished with exit code 1

Here's the version that uses just a make_one_shot_iterator():
import tensorflow as tf
from tensorflow.contrib.data import Dataset

BATCH_SIZE = 4
ITERATION_COUNT = 2
dataset = Dataset.from_tensor_slices(tf.constant([[0, 0],
                                                  [0, 1],
                                                  [1, 0],
                                                  [1, 1]], dtype=tf.float32))
batched_dataset = dataset.batch(BATCH_SIZE)

# create some graph
batched_dataset_iterator = batched_dataset.make_one_shot_iterator()
inputs = batched_dataset_iterator.get_next()
sum_placeholder = tf.placeholder_with_default(tf.reduce_sum(inputs), shape=[])
tf.add_to_collection('placeholders', sum_placeholder)
sum_variable = tf.get_variable('sum', initializer=tf.zeros(shape=[]))
assign_sum = tf.assign_add(sum_variable, sum_placeholder)
tf.add_to_collection('assigns', assign_sum)
saver = tf.train.Saver()

# run graph and save it at the end
with tf.Session() as session:
    session.run(tf.global_variables_initializer())
    inputs_feed_dict = {}
    for i in range(ITERATION_COUNT):
        inputs_sum = session.run(assign_sum, feed_dict=inputs_feed_dict)
        inputs_feed_dict[sum_placeholder] = inputs_sum
        print(inputs_sum)
    saver.save(session, 'checkpoints/haha')

# restore saved graph and run it
with tf.Session() as session:
    saver = tf.train.import_meta_graph('checkpoints/haha.meta')
    saver.restore(session, 'checkpoints/haha')
    assign_sum = tf.get_collection('assigns')[0]
    sum_placeholder = tf.get_collection('placeholders')[1]
    inputs_feed_dict = {}
    for i in range(ITERATION_COUNT):
        inputs_sum = session.run(assign_sum, feed_dict=inputs_feed_dict)
        print(inputs_sum)
        inputs_feed_dict[sum_placeholder] = inputs_sum

Result:
C:\Users\Jonathan\Miniconda3\envs\ai\python.exe "C:/Software Projects/ai/tensorflow/LayerTests.py"
2017-07-21 20:12:26.273776: W C:\tf_jenkins\home\workspace\nightly-win\M\windows-gpu\PY\36\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-07-21 20:12:26.516570: I C:\tf_jenkins\home\workspace\nightly-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:955] Found device 0 with properties:
name: GeForce GTX 670
major: 3 minor: 0 memoryClockRate (GHz) 0.98
pciBusID 0000:02:00.0
Total memory: 2.00GiB
Free memory: 1.64GiB
2017-07-21 20:12:26.516852: I C:\tf_jenkins\home\workspace\nightly-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:976] DMA: 0
2017-07-21 20:12:26.516992: I C:\tf_jenkins\home\workspace\nightly-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:986] 0:   Y
2017-07-21 20:12:26.517145: I C:\tf_jenkins\home\workspace\nightly-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 670, pci bus id: 0000:02:00.0)
4.0
8.0
2017-07-21 20:12:26.926502: I C:\tf_jenkins\home\workspace\nightly-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 670, pci bus id: 0000:02:00.0)
12.0
24.0

Process finished with exit code 0

I simplified the example some more and tried storing the iterator to feed using tf.add_to_collections() and using that to get the string handle instead of storing the string handle directly. I get a different error this time regarding initialization, but it exits fine.
import tensorflow as tf
from tensorflow.contrib.data import Dataset, Iterator

BATCH_SIZE = 2
ITERATION_COUNT = 1
dataset = Dataset.from_tensor_slices(tf.constant([[0, 0],
                                                  [0, 1],
                                                  [1, 0],
                                                  [1, 1]], dtype=tf.float32))
iterator_handle_placeholder = tf.placeholder(tf.string, shape=[])
iterator = Iterator.from_string_handle(iterator_handle_placeholder, dataset.output_types, dataset.output_shapes)
dataset_iterator = dataset.make_one_shot_iterator()

element = iterator.get_next()
v = tf.get_variable('v', initializer=tf.zeros([]))

saver = tf.train.Saver()

with tf.Session() as session:
    session.run(tf.global_variables_initializer())
    dataset_iterator_handle = session.run(dataset_iterator.string_handle())
    tf.add_to_collection('iterator_handle_placeholder', iterator_handle_placeholder)
    tf.add_to_collection('iterator', dataset_iterator)
    tf.add_to_collection('element', element)
    output = session.run(element, {iterator_handle_placeholder: dataset_iterator_handle})
    print(output)
    saver.save(session, 'checkpoints/fufu')

with tf.Session() as session:
    saver = tf.train.import_meta_graph('checkpoints/fufu.meta')
    saver.restore(session, 'checkpoints/fufu')
    iterator_handle_placeholder = tf.get_collection('iterator_handle_placeholder')[0]
    dataset_iterator_handle = session.run(tf.get_collection('iterator')[0].string_handle())
    element = tf.get_collection('element')[0]
    output = session.run(element, {iterator_handle_placeholder: dataset_iterator_handle})
    print(output)

Error message is
C:\Users\Jonathan\Miniconda3\envs\ai\python.exe "C:/Software Projects/ai/tensorflow/datasettest.py"
2017-07-21 22:15:13.371342: W C:\tf_jenkins\home\workspace\nightly-win\M\windows-gpu\PY\36\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-07-21 22:15:13.612687: I C:\tf_jenkins\home\workspace\nightly-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:955] Found device 0 with properties:
name: GeForce GTX 670
major: 3 minor: 0 memoryClockRate (GHz) 0.98
pciBusID 0000:02:00.0
Total memory: 2.00GiB
Free memory: 1.64GiB
2017-07-21 22:15:13.612996: I C:\tf_jenkins\home\workspace\nightly-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:976] DMA: 0
2017-07-21 22:15:13.613141: I C:\tf_jenkins\home\workspace\nightly-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:986] 0:   Y
2017-07-21 22:15:13.613302: I C:\tf_jenkins\home\workspace\nightly-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 670, pci bus id: 0000:02:00.0)
[ 0.  0.]
WARNING:tensorflow:Error encountered when serializing iterator.
Type is unsupported, or the types of the items don't match field type in CollectionDef.
'Iterator' object has no attribute 'name'
2017-07-21 22:15:13.999859: I C:\tf_jenkins\home\workspace\nightly-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 670, pci bus id: 0000:02:00.0)
[ 0.  0.]

Process finished with exit code 0

Hello eddiemundo,
I'm facing exactly the same issue, did you manage to solve it, any advices about how to process ?
Thank you a lot

I successfully saved and restored the model only with the iterator handle placeholder. But you should initialize a new iterator.
One strange thing is that the placeholder for the handle should have a name parameter. Without naming, the type and content is not correctly loaded when I call get_collection.

I was also struggling with this. I tried various things along the lines of what eddiemundo did, writing iterators to collections. I also tried  tf.contrib.data.make_saveable_from_iterator() (I managed to successfully save an iterator here but not get it back!). No success with this approach.
I eventually realised what youngwoo-yoon meant by "new iterator" -  you can just make an entirely new iterator in your restored model and use that instead. You still need to grab the handle from the saved model though and call iterator.string_handle() on the new iterator. Works with both one shot and initialisable iterators.
Here's a simple example with make_one_shot_iterator():
import tensorflow as tf
import numpy as np

def save(dataset):
    """
    Create graph with an Dataset and Iterator and save the model.

    There is some op that is applied to the data from the iterator.
    """
    iterator_handle = tf.placeholder(tf.string, shape=[])
    tf.add_to_collection('iterator_handle', iterator_handle)

    iterator = tf.data.Iterator.from_string_handle(
        iterator_handle,
        dataset.output_types,
        dataset.output_shapes)
    dataset_iterator = dataset.make_one_shot_iterator()
    element = iterator.get_next()

    some_op = tf.multiply(element, 0.5)
    tf.add_to_collection('some_op', some_op)
    v = tf.get_variable('v', initializer=tf.zeros([]))  # Needs a variable to save the model

    saver = tf.train.Saver()

    with tf.Session() as session:
        session.run(tf.global_variables_initializer())
        handle_val = session.run(dataset_iterator.string_handle())
        for _ in range(4):
            print(session.run(some_op,
                feed_dict={iterator_handle: handle_val}))
        saver.save(session, 'checkpoints/fufu')

def restore(dataset):
    """Restore the model from file and pass some new data through it"""
    with tf.Session() as session:
        saver = tf.train.import_meta_graph('checkpoints/fufu.meta')
        saver.restore(session, 'checkpoints/fufu')
        iterator_handle = tf.get_collection('iterator_handle')[0]
        # Make new iterator
        iterator = dataset.make_one_shot_iterator()
        new_handle = session.run(iterator.string_handle())
        element = iterator.get_next()
        some_op = tf.get_collection('some_op')[0]
        for _ in range(4):
            print(session.run(some_op, {iterator_handle: new_handle}))

if __name__ == '__main__':

    raw_data = np.array([[0, 0],
                     [0, 1],
                     [1, 0],
                     [1, 1]])
    dataset1 = tf.data.Dataset.from_tensor_slices(tf.constant(raw_data, dtype=tf.float32))
    dataset2 = tf.data.Dataset.from_tensor_slices(tf.constant(raw_data * 2, dtype=tf.float32))

    save(dataset1)

    # Restore works with any data of the same shape in a tf.data.Dataset
    # To use different shaped data use initialisable iterator
    restore(dataset1)
    restore(dataset2)

Here's another example with initialisable iterators.
# Tensorflow 1.8.0
import tensorflow as tf
import numpy as np

def train(train_dataset, test_dataset):
    """
    Create graph with an Dataset and Iterator and save the model.

    There is some op that is applied to the data from the iterator.
    """
    iterator_handle = tf.placeholder(tf.string, shape=[])
    tf.add_to_collection('iterator_handle', iterator_handle)

    iterator = tf.data.Iterator.from_string_handle(
        iterator_handle,
        train_dataset.output_types,
        train_dataset.output_shapes)
    train_iter = train_dataset.make_initializable_iterator()
    test_iter = test_dataset.make_initializable_iterator()
    element = iterator.get_next()

    some_op = tf.multiply(element, 0.5)
    tf.add_to_collection('some_op', some_op)
    v = tf.get_variable('v', initializer=tf.zeros([]))

    saver = tf.train.Saver()

    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())

        train_handle = sess.run(train_iter.string_handle())
        test_handle = sess.run(test_iter.string_handle())

        # Run data iterator initialisation
        sess.run(train_iter.initializer)
        sess.run(test_iter.initializer)

        # "Training"
        print("Training")
        while True:
            try:
                print(sess.run(some_op,
                    feed_dict={iterator_handle: train_handle}))
            except tf.errors.OutOfRangeError:
                break

        # "Test evaluation"
        print("Testing")
        while True:
            try:
                print(sess.run(some_op,
                    feed_dict={iterator_handle: test_handle}))
            except tf.errors.OutOfRangeError:
                break

        saver.save(sess, 'checkpoints/fufu')

def eval(dataset):
    """Restore the model from file and pass some new data through it"""
    with tf.Session() as sess:
        saver = tf.train.import_meta_graph('checkpoints/fufu.meta')
        saver.restore(sess, 'checkpoints/fufu')
        iterator_handle = tf.get_collection('iterator_handle')[0]
        # Make new iterator
        iterator = dataset.make_one_shot_iterator()
        new_handle = sess.run(iterator.string_handle())
        # Don't need to call iterator.get_next() again as `some_op` will use
        # restored `element`
        some_op = tf.get_collection('some_op')[0]

        # "Further evaluation"
        print("More testing")
        while True:
            try:
                print(sess.run(some_op,
                    feed_dict={iterator_handle: new_handle}))
            except tf.errors.OutOfRangeError:
                break

if __name__ == '__main__':

    train_dataset = tf.data.Dataset.from_tensor_slices(
        tf.constant(np.random.randint(0, 100, (5, 2)), dtype=tf.float32))
    test_dataset = tf.data.Dataset.from_tensor_slices(
        tf.constant(np.random.randint(0, 100, (2, 2)), dtype=tf.float32))

    train(train_dataset, test_dataset)

    # Now want to evaluate the results of another test dataset
    another_test_dataset = tf.data.Dataset.from_tensor_slices(
        tf.constant(np.random.randint(0, 100, (4, 2)), dtype=tf.float32))
    eval(another_test_dataset)

Your way of saving an iterator handle and then feeding it back in is the only way that I can get to work. Thanks!
Is this the most efficient way to do this? It seems like having to do an op lookup from a handle for every train step is less efficient than just grafting in your new iterator one time during graph setup.
What made the most sense to me would have been to save a reinitializable iterator itself into the metagraph, and then to restore that iterator and call sess.run(reinitializable_iterator.make_initializer(my_new_dataset)). That only needs to happen once per iterator initialization instead of every train set.  But, I was unable to save an reinitializable iterator to the metagraph and then properly restore it. After a few hours of trying, I gave up and switched to feedable.
I think that attaching a new Dataset iterator to a restored model is the most common use case for metagraphs, so I think that these examples you're giving here should be in the docs. If it is indeed the best way to do this.

Glad someone found the examples helpful. I was going a bit nuts not being able to find an example of what I thought would be a fairly common use case.
I have no idea RE efficiency of this. I agree that using a reinitializable_iterator seems much more elegant, but like you I haven't managed to get it working with a restored model. The op lookup from a handle is probably OK (feels like using get_variable() elsewhere in the graph to get some weights), but this is mere conjecture.
The great irony of all this is I've not noticed any difference between using tf.data.Dataset + iterators and just passing in data via a feed_dict like most of the tutorials do. I'm assuming there's an amount of data where using iterators becomes better but I've not hit it yet.

Since we both encountered the same difficulty and solved it in the same way, I think it's a good enough hint that the docs should be updated. Filed a new bug.

Thank you, @annarailton for sharing that complete example.
In addition to training and then loading the saved model in the development phase, I want to load and continue training the saved model.
Everything works fine, but there is a small problem here. Whenever I restore the previous model and resume training, operations related to creating new datasets and iterators are added to the existing graph. This makes the graph bigger and bigger every time that I load it for further training. To be able to see this problem you need to visualize the computation graph in Tensorboard.
The following is a modified version of your code to make the problem clear.  Any thoughts or suggestions?
# Tensorflow 1.8.0
import tensorflow as tf
import numpy as np
import os
import shutil

def train(train_dataset, test_dataset):
    """
    Create graph with an Dataset and Iterator and save the model.

    There is some op that is applied to the data from the iterator.
    """
    iterator_handle = tf.placeholder(tf.string, shape=[])
    tf.add_to_collection('iterator_handle', iterator_handle)

    iterator = tf.data.Iterator.from_string_handle(
        iterator_handle,
        train_dataset.output_types,
        train_dataset.output_shapes)
    train_iter = train_dataset.make_initializable_iterator()
    test_iter = test_dataset.make_initializable_iterator()
    element = iterator.get_next()

    v = tf.get_variable(name='v', initializer=tf.zeros(shape=(1, 2)))

    # to use when saving summaries
    global_step = tf.Variable(0, name='global_step', trainable=False, dtype=tf.int32)
    increament_global_step = tf.assign(global_step, global_step + 1)
    global_step = global_step + 1
    tf.add_to_collection('increament_global_step', increament_global_step)

    some_op = tf.assign(v,  v + tf.abs(element))
    tf.add_to_collection('some_op', tf.reduce_sum(some_op))

    tf.summary.scalar('v_sum', tf.reduce_sum(v))
    tf.summary.scalar('some_op', tf.reduce_mean(some_op))
    merged_summary = tf.summary.merge_all()
    tf.add_to_collection('merged_summary', merged_summary)

    writer = tf.summary.FileWriter('checkpoints', graph=tf.get_default_graph())
    saver = tf.train.Saver()

    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())

        train_handle = sess.run(train_iter.string_handle())
        test_handle = sess.run(test_iter.string_handle())

        # Run data iterator initialisation
        sess.run(train_iter.initializer)
        sess.run(test_iter.initializer)

        # "Training"
        print("Training")
        while True:
            try:
                [op, summary_values, g_step] = sess.run([some_op, merged_summary, increament_global_step],
                                                feed_dict={iterator_handle: train_handle})
                writer.add_summary(summary_values, global_step=g_step)
                print(op)
            except tf.errors.OutOfRangeError:
                break

        # "Test evaluation"
        print("Testing")
        while True:
            try:
                print(sess.run(some_op,
                    feed_dict={iterator_handle: test_handle}))
            except tf.errors.OutOfRangeError:
                break

        saver.save(sess, 'checkpoints/fufu')

def eval(dataset):
    """Restore the model from file and pass some new data through it"""
    with tf.Session() as sess:
        saver = tf.train.import_meta_graph('checkpoints/fufu.meta')
        saver.restore(sess, 'checkpoints/fufu')
        iterator_handle = tf.get_collection('iterator_handle')[0]
        # Make new iterator
        iterator = dataset.make_one_shot_iterator()
        new_handle = sess.run(iterator.string_handle())
        # Don't need to call iterator.get_next() again as `some_op` will use
        # restored `element`
        some_op = tf.get_collection('some_op')[0]

        # "Further evaluation"
        print("More testing")
        while True:
            try:
                print(sess.run(some_op,
                    feed_dict={iterator_handle: new_handle}))
            except tf.errors.OutOfRangeError:
                break

def resume_training(train_dataset, test_dataset):
    """Restore the model from file and pass some new data through it
     for further training """
    with tf.Session() as sess:
        saver = tf.train.import_meta_graph('checkpoints/fufu.meta')
        saver.restore(sess, 'checkpoints/fufu')
        iterator_handle = tf.get_collection('iterator_handle')[0]
        some_op = tf.get_collection('some_op')[0]
        increament_global_step = tf.get_collection('increament_global_step')[0]
        merged_summary = tf.get_collection('merged_summary')[0]

        writer = tf.summary.FileWriter('checkpoints',
                                       graph=tf.get_default_graph())

        # Make new iterators and handles
        train_iter = train_dataset.make_initializable_iterator()
        test_iter = test_dataset.make_initializable_iterator()

        train_handle = sess.run(train_iter.string_handle())
        test_handle = sess.run(test_iter.string_handle())

        # Further training the model using new datasets (which may be different from original ones)
        print("Resume training ...")

        train_handle = sess.run(train_iter.string_handle())
        test_handle = sess.run(test_iter.string_handle())

        # Run data iterator initialisation
        sess.run(train_iter.initializer)
        sess.run(test_iter.initializer)

        # "Training"
        print("Training")
        while True:
            try:
                [op, summary_values, g_step] = sess.run([some_op, merged_summary, increament_global_step],
                                                feed_dict={
                                                    iterator_handle: train_handle})
                writer.add_summary(summary_values, global_step=g_step)
                print(op)
            except tf.errors.OutOfRangeError:
                break

        # "Test evaluation"
        print("Testing")
        while True:
            try:
                print(sess.run(some_op,
                               feed_dict={iterator_handle: test_handle}))
            except tf.errors.OutOfRangeError:
                break

        saver.save(sess, 'checkpoints/fufu')

if __name__ == '__main__':

    flag = 3  # 1 for train, 2 for test, 3 to resume training
    if flag == 1:
        # delete existing saved models and summary files
        if os.path.exists('checkpoints'):
            shutil.rmtree('checkpoints')
        train_dataset = tf.data.Dataset.from_tensor_slices(
            tf.constant(np.random.randint(0, 100, (5, 2)), dtype=tf.float32))
        test_dataset = tf.data.Dataset.from_tensor_slices(
            tf.constant(np.random.randint(0, 100, (2, 2)), dtype=tf.float32))

        train(train_dataset, test_dataset)
    elif flag == 2:
        # Now want to evaluate the results of another test dataset
        another_test_dataset = tf.data.Dataset.from_tensor_slices(
            tf.constant(np.random.randint(0, 100, (4, 2)), dtype=tf.float32))
        eval(another_test_dataset)

    elif flag == 3:
        # Load and fine-tune the saved model using new data
        another_train_dataset = tf.data.Dataset.from_tensor_slices(
            tf.constant(np.random.randint(0, 100, (10, 2)), dtype=tf.float32))
        another_test_dataset = tf.data.Dataset.from_tensor_slices(
            tf.constant(np.random.randint(0, 100, (8, 2)), dtype=tf.float32))

        resume_training(another_train_dataset, another_test_dataset)


Updated Java getting started steps with sample app

Closes #
Updated java getting started steps in the guide with example code.

[x] Signed the InfluxData CLA
(if necessary)
[x] Rebased/mergeable

Hi @sanderson can you please review these changes? Thanks!


http error when attempting to push to remote repository

When attempting to push from local to remite repository, the following error comes up.
HTTP_ERROR : Address family not supported by protocol - socket(2) for "localhost" port 3000
The full verbose shown at the command line prompt shows as follows.
user@machine ~/D/rtvn-joomla-component-to-collect-leads> git push -u origin master
Counting objects: 2, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 243 bytes | 0 bytes/s, done.
Total 2 (delta 0), reused 0 (delta 0)

Notifying Redmine about changes to this repository : 'rtvn-joomla-component-to-collect-leads/' ...
HTTP_ERROR : Address family not supported by protocol - socket(2) for "localhost" port 3000

Looking for additional global post-receive hooks...
  - Global hooks found : ["hooks/post-receive.d/mail_notifications"]

Looking for additional local post-receive hooks...
  - No local hooks found

Calling additional post-receive hooks...
  - Executing extra hook 'hooks/post-receive.d/mail_notifications'
    multimailhook is disabled

To<EMAIL_ADDRESS>   5883232..d29bd65  master -> master
Branch master set up to track remote branch master from origin.

I'm not sure if the error in question is preventing the successful completion of the operation, but the fact of the matter is that the 2 files added to the local copy do not upload to the remote repository and therefore the plugin does not show any files on the Repository tab.

Sorted.
Running
ruby bin/rails server -e production -b <IP_ADDRESS>
from the redmine root dir did the trick.

Hi! check your /etc/hosts file ;)


Improved readability of access-control.md

@JayasuriyanRP Please sign the Contributor License Agreement!
Click here to manually synchronize the status of this Pull Request.
See the FAQ for frequently asked questions.

@JayasuriyanRP Thank you for signing the Contributor License Agreement!

Thank you!


how to access java public field

it read the java-interop doc:
https://eta-lang.org/docs/eta-concepts/java-interop/java-ffi
it seems the public field access case is missing.
it's a very common case, (System.out...)

In fact the public case is the only case that works according to the " Importing Instance Fields " section. Honestly, it's a bit confusing so we'll use System.out as the example instead.

i use the following code to run println method, it report those error,
while i use IO type, it will success: [
Exception in thread "main" java.lang.NoClassDefFoundError:
at main.Main$$wa.applyV(Main.hs:15)
at main.Main$main1.applyV(Main.hs)
at main.Main$DZCmain.applyV(Main.hs)
at eta.runtime.stg.Closures$EvalLazyIO.enter(Closures.java:104)
at eta.runtime.stg.Capability.schedule(Capability.java:157)
at eta.runtime.stg.Capability.scheduleClosure(Capability.java:102)
at eta.runtime.Runtime.evalLazyIO(Runtime.java:189)
at eta.runtime.Runtime.main(Runtime.java:182)
at eta.main.main(Unknown Source)
Caused by: java.lang.ClassNotFoundException:
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
module Main where

import Java

data System = System @java.lang.System
  deriving Class

data PrintStream = PrintStream @java.io.PrintStream
  deriving Class

foreign import java unsafe "@static @field java.lang.System.out" getOut :: Java a PrintStream
foreign import java unsafe "println" println :: PrintStream -> String -> Java a ()
-- foreign import java unsafe "println" println :: PrintStream -> String -> IO ()

main :: IO ()
main = do
  out <- java $ getOut
  java $ println out "Hello World"

@clojurians-org Can you report this as a separate issue since this looks like a bug? Thanks!