Exchange Mailbox creation  problems

Hi, i want to enable mailbox on certain users of my AD (no all will have mailbox) so far this library its been working great but cant seem to do this, i have the following in my code:
// im ommiting the query of ldap for the user as its fine as it can be. ($username)
$server = $ldap->buildLdapQuery()
            ->from(LdapObjectType::EXCHANGE_SERVER)
            ->where(['guid' => $server_guid])
            ->getLdapQuery()
            ->getSingleResult();

thr result is this (ok so far):
Object(LdapTools\Object\LdapObject)#161 (4) {
  ["functions":protected]=>
  array(6) {
    [0]=>
    string(3) "has"
    [1]=>
    string(3) "get"
    [2]=>
    string(3) "set"
    [3]=>
    string(6) "remove"
    [4]=>
    string(3) "add"
    [5]=>
    string(5) "reset"
  }
  ["type":protected]=>
  string(14) "ExchangeServer"
  ["attributes":protected]=>
  array(6) {
    ["name"]=>
    string(9) "SRVMAILB6"
    ["version"]=>
    array(1) {
      [0]=>
      string(47) "Exchange Server 2010 SP3 (Build 14.03.0123.004)"
    }
    ["dn"]=>
    string(206) "CN=SRVMAILB6,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=GobiernoYucatan,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=gobierno1,DC=yucatan,DC=gob,DC=mx"
    ["guid"]=>
    array(1) {
      [0]=>
      string(36) "4bfcf0b7-cb1b-44f1-a3df-7ff6b78c9711"
    }
    ["legacyExchangeDn"]=>
    string(110) "/o=GobiernoYucatan/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Configuration/cn=Servers/cn=SRVMAILB6"
    ["roles"]=>
    array(1) {
      [0]=>
      array(1) {
        [0]=>
        string(15) "MailboxDatabase"
      }
    }
  }
  ["batches":protected]=>
  object(LdapTools\BatchModify\BatchCollection)#181 (2) {
    ["batches":protected]=>
    array(0) {
    }
    ["dn":protected]=>
    string(206) "CN=SRVMAILB6,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=GobiernoYucatan,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=gobierno1,DC=yucatan,DC=gob,DC=mx"
  }
}

then the database:
$database = $ldap->buildLdapQuery()
            ->from(LdapObjectType::EXCHANGE_DATABASE)
            ->where(['guid' => $database_guid])
            ->getLdapQuery()
            ->getSingleResult();

again everithing is fine
object(LdapTools\Object\LdapObject)#112 (4) {
  ["functions":protected]=>
  array(6) {
    [0]=>
    string(3) "has"
    [1]=>
    string(3) "get"
    [2]=>
    string(3) "set"
    [3]=>
    string(6) "remove"
    [4]=>
    string(3) "add"
    [5]=>
    string(5) "reset"
  }
  ["type":protected]=>
  string(16) "ExchangeDatabase"
  ["attributes":protected]=>
  array(3) {
    ["name"]=>
    string(8) "BDYUCVIP"
    ["dn"]=>
    string(207) "CN=BDYUCVIP,CN=Databases,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=GobiernoYucatan,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=gobierno1,DC=yucatan,DC=gob,DC=mx"
    ["guid"]=>
    array(1) {
      [0]=>
      string(36) "262887b1-3531-4f7b-80b1-05cb3187181b"
    }
  }
  ["batches":protected]=>
  object(LdapTools\BatchModify\BatchCollection)#109 (2) {
    ["batches":protected]=>
    array(0) {
    }
    ["dn":protected]=>
    string(207) "CN=BDYUCVIP,CN=Databases,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=GobiernoYucatan,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=gobierno1,DC=yucatan,DC=gob,DC=mx"
  }
}

then i try the sample code to enable the mail box
$mailbox = $ldap->createLdapObject(LdapObjectType::EXCHANGE_MAILBOX_USER)
            ->with([
                'username' => $username,
                'password' => $password,
                'mailboxDatabase' => $database,
                'mailboxServer' => $server
            ])
            ->in( Config::get('LDAP_BASE_DN'))
            ->execute();

and here is where nightmare begins, at first i start gettin errors when the class was trying to parse certain parameters toLdap like so:
"Unable to convert value \"Default Policy\" to a objectGuid for attribute \"recipientPolicies\""
"Unable to convert value \"All Mailboxes(VLV)\" to a dn for attribute \"showInAddressBooks\"
"Unable to convert value \"Default Role Assignment Policy\" to a dn for attribute \"rbacPolicy\""

all comming from the same place
"#0 D:\\SOFTWARE\\Xampp\\htdocs\\GIT\\buzones-correo\\vendor\\ldaptools\\ldaptools\\src\\LdapTools\\AttributeConverter\\ConvertValueToDn.php(66): LdapTools\\AttributeConverter\\ConvertValueToDn->getAttributeFromLdapQuery('Default Role As...', 'dn')",
                "#1 D:\\SOFTWARE\\Xampp\\htdocs\\GIT\\buzones-correo\\vendor\\ldaptools\\ldaptools\\src\\LdapTools\\Resolver\\BaseValueResolver.php(234): LdapTools\\AttributeConverter\\ConvertValueToDn->toLdap('Default Role As...')",
                "#2 D:\\SOFTWARE\\Xampp\\htdocs\\GIT\\buzones-correo\\vendor\\ldaptools\\ldaptools\\src\\LdapTools\\Resolver\\BaseValueResolver.php(154): LdapTools\\Resolver\\BaseValueResolver->doConvertValues('rbacPolicy', Array, 'toLdap', NULL)",
                "#3 D:\\SOFTWARE\\Xampp\\htdocs\\GIT\\buzones-correo\\vendor\\ldaptools\\ldaptools\\src\\LdapTools\\Resolver\\AttributeValueResolver.php(92): LdapTools\\Resolver\\BaseValueResolver->getConvertedValues('Default Role As...', 'rbacPolicy', 'toLdap')",
                "#4 D:\\SOFTWARE\\Xampp\\htdocs\\GIT\\buzones-correo\\vendor\\ldaptools\\ldaptools\\src\\LdapTools\\Resolver\\AttributeValueResolver.php(71): LdapTools\\Resolver\\AttributeValueResolver->convert(Array)",
                "#5 D:\\SOFTWARE\\Xampp\\htdocs\\GIT\\buzones-correo\\vendor\\ldaptools\\ldaptools\\src\\LdapTools\\Hydrator\\ArrayHydrator.php(313): LdapTools\\Resolver\\AttributeValueResolver->toLdap()",
                "#6 D:\\SOFTWARE\\Xampp\\htdocs\\GIT\\buzones-correo\\vendor\\ldaptools\\ldaptools\\src\\LdapTools\\Hydrator\\ArrayHydrator.php(174): LdapTools\\Hydrator\\ArrayHydrator->convertValuesToLdap(Array)",
                "#7 D:\\SOFTWARE\\Xampp\\htdocs\\GIT\\buzones-correo\\vendor\\ldaptools\\ldaptools\\src\\LdapTools\\Hydrator\\OperationHydrator.php(114): LdapTools\\Hydrator\\ArrayHydrator->hydrateToLdap(Array)",
                "#8 D:\\SOFTWARE\\Xampp\\htdocs\\GIT\\buzones-correo\\vendor\\ldaptools\\ldaptools\\src\\LdapTools\\Hydrator\\OperationHydrator.php(80): LdapTools\\Hydrator\\OperationHydrator->hydrateAddOperation(Object(LdapTools\\Operation\\AddOperation))",
                "#9 D:\\SOFTWARE\\Xampp\\htdocs\\GIT\\buzones-correo\\vendor\\ldaptools\\ldaptools\\src\\LdapTools\\Hydrator\\OperationHydrator.php(56): LdapTools\\Hydrator\\OperationHydrator->hydrateOperation(Object(LdapTools\\Operation\\AddOperation))",
                "#10 D:\\SOFTWARE\\Xampp\\htdocs\\GIT\\buzones-correo\\vendor\\ldaptools\\ldaptools\\src\\LdapTools\\Object\\LdapObjectCreator.php(272): LdapTools\\Hydrator\\OperationHydrator->hydrateToLdap(Object(LdapTools\\Operation\\AddOperation))",
                "#11 D:\\SOFTWARE\\Xampp\\htdocs\\GIT\\buzones-correo\\vendor\\ldaptools\\ldaptools\\src\\LdapTools\\Object\\LdapObjectCreator.php(252): LdapTools\\Object\\LdapObjectCreator->getAddOperation()",
                "#12 D:\\SOFTWARE\\Xampp\\htdocs\\GIT\\buzones-correo\\App\\Directorio\\Controllers\\DirectorioController.php(210): LdapTools\\Object\\LdapObjectCreator->execute()"

as i wasnt able to determine what was wrong here i modified the method convert of AttributeValueResolver.php like this:
protected function convert(array $attributes, $toLdap = true)
    {
        $direction = ($toLdap) ? 'toLdap' : 'fromLdap';

        foreach ($attributes as $attribute => $values) {
            // No converter, but the value should still be encoded.
            if($attribute == 'rbacPolicy' ||
                $attribute == 'recipientPolicies' ||
                $attribute == 'showInAddressBooks'
                ) { continue; }

and then i got this:
"Unable to determine msExchVersion version number for build \"06.05\" and attribute \"version\""

i know my server is an exchange 2010 but the auto parameter seems to get the version wrong, again i modified for debug the file ConvertExchangeObjectVersion.php specificly the $buildMap array to:
protected $buildMap =[
        '08' => '2007',
        '14' => '2010',
        '06.05' => '2010',
        '15.00' => '2013',
        '15.01' => '2016',
    ];

just to get another one:
"To send a password to AD you need to enable either TLS or SSL in your configuration."

at:
                "#0 D:\\SOFTWARE\\Xampp\\htdocs\\GIT\\buzones-correo\\vendor\\ldaptools\\ldaptools\\src\\LdapTools\\AttributeConverter\\EncodeWindowsPassword.php(32): LdapTools\\AttributeConverter\\EncodeWindowsPassword->validateConfiguration()",
                "#1 D:\\SOFTWARE\\Xampp\\htdocs\\GIT\\buzones-correo\\vendor\\ldaptools\\ldaptools\\src\\LdapTools\\Resolver\\BaseValueResolver.php(234): LdapTools\\AttributeConverter\\EncodeWindowsPassword->toLdap('')",
                "#2 D:\\SOFTWARE\\Xampp\\htdocs\\GIT\\buzones-correo\\vendor\\ldaptools\\ldaptools\\src\\LdapTools\\Resolver\\BaseValueResolver.php(154): LdapTools\\Resolver\\BaseValueResolver->doConvertValues('password', Array, 'toLdap', NULL)",
                "#3 D:\\SOFTWARE\\Xampp\\htdocs\\GIT\\buzones-correo\\vendor\\ldaptools\\ldaptools\\src\\LdapTools\\Resolver\\AttributeValueResolver.php(96): LdapTools\\Resolver\\BaseValueResolver->getConvertedValues('', 'password', 'toLdap')",
                "#4 D:\\SOFTWARE\\Xampp\\htdocs\\GIT\\buzones-correo\\vendor\\ldaptools\\ldaptools\\src\\LdapTools\\Resolver\\AttributeValueResolver.php(71): LdapTools\\Resolver\\AttributeValueResolver->convert(Array)",
                "#5 D:\\SOFTWARE\\Xampp\\htdocs\\GIT\\buzones-correo\\vendor\\ldaptools\\ldaptools\\src\\LdapTools\\Hydrator\\ArrayHydrator.php(313): LdapTools\\Resolver\\AttributeValueResolver->toLdap()",
                "#6 D:\\SOFTWARE\\Xampp\\htdocs\\GIT\\buzones-correo\\vendor\\ldaptools\\ldaptools\\src\\LdapTools\\Hydrator\\ArrayHydrator.php(174): LdapTools\\Hydrator\\ArrayHydrator->convertValuesToLdap(Array)",
                "#7 D:\\SOFTWARE\\Xampp\\htdocs\\GIT\\buzones-correo\\vendor\\ldaptools\\ldaptools\\src\\LdapTools\\Hydrator\\OperationHydrator.php(114): LdapTools\\Hydrator\\ArrayHydrator->hydrateToLdap(Array)",
                "#8 D:\\SOFTWARE\\Xampp\\htdocs\\GIT\\buzones-correo\\vendor\\ldaptools\\ldaptools\\src\\LdapTools\\Hydrator\\OperationHydrator.php(80): LdapTools\\Hydrator\\OperationHydrator->hydrateAddOperation(Object(LdapTools\\Operation\\AddOperation))",
                "#9 D:\\SOFTWARE\\Xampp\\htdocs\\GIT\\buzones-correo\\vendor\\ldaptools\\ldaptools\\src\\LdapTools\\Hydrator\\OperationHydrator.php(56): LdapTools\\Hydrator\\OperationHydrator->hydrateOperation(Object(LdapTools\\Operation\\AddOperation))",
                "#10 D:\\SOFTWARE\\Xampp\\htdocs\\GIT\\buzones-correo\\vendor\\ldaptools\\ldaptools\\src\\LdapTools\\Object\\LdapObjectCreator.php(272): LdapTools\\Hydrator\\OperationHydrator->hydrateToLdap(Object(LdapTools\\Operation\\AddOperation))",
                "#11 D:\\SOFTWARE\\Xampp\\htdocs\\GIT\\buzones-correo\\vendor\\ldaptools\\ldaptools\\src\\LdapTools\\Object\\LdapObjectCreator.php(252): LdapTools\\Object\\LdapObjectCreator->getAddOperation()",
                "#12 D:\\SOFTWARE\\Xampp\\htdocs\\GIT\\buzones-correo\\App\\Directorio\\Controllers\\DirectorioController.php(210): LdapTools\\Object\\LdapObjectCreator->execute()"

at this point i feel kind of stuck im not sure that enabling ssl will do anything besides the pain of dealing with the server people im not even sure it will work, besides mi ideal scenario is that the password for the mailbox will be the same as the ldap user password, and im not able to query that, so far i was passing an empty string.
can someone tell me if this is a bug, something im doing wrong or anything else please

I have not had time to focus on this library lately, so I'm not sure how much I can assist. But there are quite a few things that come to mind.

Are you trying to mail enable an already existing user? If that is the case, then this library does not yet support that functionality. It can create new AD users as mail enabled Exchange users though.
What language was Exchange installed as on the server end? English? I feel like the default queries might be failing to locate stuff if this is not the case. Just a hunch though.
It's actually an AD requirement that if you need to do an operation that involves a password, it will force SSL / TLS. Although this can be disabled via the registry on a DC, no one administering AD should EVER do that. You can enable SSL/TLS for this library and then just disable certificate validation (not recommended, but better than no SSL at all). See the config documentation for how to get that working.

Another thought, it's also possible the policies were renamed. You can query for them the same way as the Exchange Database, and then set them on the user when you create them.

I have not had time to focus on this library lately, so I'm not sure how much I can assist. But there are quite a few things that come to mind.

Are you trying to mail enable an already existing user? If that is the case, then this library does not yet support that functionality. It can create new AD users as mail enabled Exchange users though.
What language was Exchange installed as on the server end? English? I feel like the default queries might be failing to locate stuff if this is not the case. Just a hunch though.
It's actually an AD requirement that if you need to do an operation that involves a password, it will force SSL / TLS. Although this can be disabled via the registry on a DC, no one administering AD should EVER do that. You can enable SSL/TLS for this library and then just disable certificate validation (not recommended, but better than no SSL at all). See the config documentation for how to get that working.

Yes that was my goal, but what about deleting the account an recreating it as mail enabled?
is in english
3 and extra. ill give a shot

anyway thaks for the tips even if im not able to get this to work and fallback to the powershell script this library was a great addition to my project, again thanks

Deleting the account, saving existing user properties, then re-applying to a new mailbox user while creating them would work. Though if the user has individual permissions applied anywhere (like on a fileserver or something), then those would break.
My guess is that your Exchange policy names are different from the defaults, or have been changed. In that instance you would need to explicitly search for and use them. Like for the RBAC and Recipient Policy:
$recipientPolicies = $ldap->buildLdapQuery()
            ->from(LdapObjectType::EXCHANGE_RECIPIENT_POLICY)
            ->getLdapQuery()
            ->getResult();

$rbacPolicies = $ldap->buildLdapQuery()
            ->from(LdapObjectType::EXCHANGE_RBAC_POLICY)
            ->getLdapQuery()
            ->getResult();

Then when creating the user you would assign the RBAC policy to the rbacPolicy attribute and the recipient policy to recipientPolicies.
Regarding the Exchange Version, it would seem that you still have an Exchange Server 2003 system in your environment, or it was never removed properly and is still in AD. As version 6.5 is for Exchange 2003. You could get around that by specifically referencing the Exchange Version when creating the user: ['version' => 'v2010'].
You can verify the version it pulls from your environment automagically by running this:
var_dump($query->select('serialNumber')
            ->where(['objectClass' => 'msExchExchangeServer'])
            ->andWhere($query->filter()->present('serverRole'))
            ->setBaseDn('%_configurationnamingcontext_%')
            ->setSizeLimit(1)
            ->getLdapQuery()
            ->getSingleScalarOrNullResult());

That's where it's finding some kind of Exchange Server 2003 info.


build: bump angular

@angular/cli       18.1.0 -> 18.1.1
@angular/core      18.1.0 -> 18.1.1
@angular/cdk       18.1.0 -> 18.1.1
@angular/material  18.1.0 -> 18.1.1
PR Checklist

[ ] Unit Tests have been added for new changes
[ ] API tests have been updated if applicable
[ ] All commented code has been removed
[ ] If you've added a dependency, you've ensured license is compatible with Apache 2.0 and clearly outlined the added dependency.

What are you changing?

Anything the reviewer should know when reviewing this PR?
If the there are associated PRs in other repositories, please link them here (i.e. open-amt-cloud-toolkit/repo#365 )

:tada: This PR is included in version 3.12.2 :tada:
The release is available on:

GitHub release
v3.12.2

Your semantic-release bot :package::rocket:


Improve download speed of records by the CLI and jdbc client

The sequential read by the client makes the download really slow. Split it into two threads, with one thread downloading the data and buffering while the other thread renders it to stdout (CLI) or returns it to the user (jdbc)

Thanks for contributing this feature. It will be useful to many people.
I added a few high level comments. Can you split the CLI and JDBC changes into separate commits, or even separate pull requests if that's easier? They are logically separate, so that will make them easier to review.

Keeping the CLI and JDBC changes in the same PR as the changes are identical and after incorporating review comments they are impacting fewer files.

Hi @electrum,
TestJdbcWarnings.testExecuteQueryWarnings() is failing as we are processing multiple pages at the same time. The warnings of the current page are cleared by the next page at PrestoResultSet.computeNext() before the resultset.getWarnings() is called on the current page.
A simple solution might be getting rid of warningsManager.clearWarning() in PrestoResultSet.computeNext() else, we will have to maintain a different data structure to map the row-index range with the warnings it produced and, set and unset them accordingly.
Can you please suggest how should we go about it, how should we handle it without adding extra overhead.

Hi @electrum,
Did you get time to look into this?
Can you please provide your suggestion.
Thanks,
Shashwat.

#1640 should fix the warnings issue

Apologies for a long delay in incorporating the review comments. I have updated the pull request, please take a look.

Merged, thanks!


Upload multiple mp4 files to same task

Actions before raising this issue

[X] I searched the existing issues and did not find anything similar.
[X] I read/searched the docs

Is your feature request related to a problem? Please describe.
I connected CVAT to shared file storge and that works properly.
When I tried to upload a folder with several mp4 file I got this error.

I read that another issues that posted in 2020 and 2021 and 2022 that asked about that and you said that it will be implemented in next versions.
Describe the solution you'd like
when upload folder with multiple videos it will works .
Describe alternatives you've considered
No response
Additional context
There is a way to create new task that contains multiple mp4 files ??

I read that another issues that posted in 2020 and 2021 and 2022 that asked about that and you said that it will be implemented in next versions.

I do not think that is true. That is againist core concepts. One task can not contain more than one videofile.


creates downloadable zip file of all patients

Provide the Summary of the Results also.

@satyakirankumar There isn't really a good way for us to do that, as there is no standard for displaying this information.


Add Dollhouse, Add The Lone Gunmen (TV Series)

Checklist for submitting a pull request to Movies For Hackers:

[x] I have carefully read and comply with the Contributing Guidelines of this repo.
[x] I have searched the PRs of this repo and believe that this is not a duplicate.

Title: Dollhouse
Link: https://www.imdb.com/title/tt1135300/
Genre: Drama, Mystery, Sci-Fi
Rating: 7.7

Description: A futuristic laboratory has erased the identities of lost young people, and now imprints them with the temporary identities they need to fulfill assignments for clients.
#203

Hi there @aicouto! Please create two different PRs for your suggestions. Thanks!

ok, one by one:

Dollhouse

Closing due to no response from the PR's author.


WIP Fix SS4 AssetStore compat in FixtureContext

Correctly create published files, and delete them via the AssetStore API rather than unlink(). Since the merge of https://github.com/silverstripe/silverstripe-cms/pull/1381 cms Behat builds are broken due to this. Figured I'd open the PR before its ready to avoid the usual suspects (@dhensby and @tractorcow) toiling over the same problem :)
Again: This isn't ready to merge, work in progress.

I should not have merged that... sorry, I read the last line just as it was too late. :)


Latest GOG Galaxy update breaks Lauhdutin

Short description of the issue
Games won't load after GOG got updated, error message comes up
Lauhdutin version
3.0.1
Rainmeter version
4.2.0 beta r3029 64-bit (Feb 8 2018)
Windows version
Windows 10 Pro 64-bit (build 16299)
Error messages

DBUG (20:18:08.147) : Initializing skin
DBUG (20:18:08.154) : Loading translation file
DBUG (20:18:08.164) : Initializing
DBUG (20:18:08.896) : Detecting games
DBUG (20:18:08.909) : Num platforms:    4
DBUG (20:18:08.915) : Registering platform processes
DBUG (20:18:08.920) : - 2 = Steam.exe
DBUG (20:18:08.926) : - 3 = Battle.net.exe
DBUG (20:18:08.932) : - 4 = GalaxyClient.exe
DBUG (20:18:08.938) :  Windows shortcut = true
DBUG (20:18:08.944) :  Steam = true
DBUG (20:18:08.950) :  Blizzard Battle.net = true
ERRO (20:18:08.956) Lauhdutin.-.3.0.1\Main.ini: Error: init.lua:10: The path to GOG Galaxy's ProgramData directory is not valid.
Steps to reproduce the issue
Loading my computer and or refreshing rainmeter

Do you have any game installed on GOG Galaxy? Seems like last update removes index.db from my ProgramData GOG folder, but I don't have any game installed so I don't know if they delete index.db if you don't have any game installed or they will handle library with ownership.db from now...

This should be fixed in 3.0.2, which was just released.

Awesome! Is it possible to update without having to go through all the settings?

All of your games and settings are stored in<EMAIL_ADDRESS>and<EMAIL_ADDRESS>respectively. Neither of those are included in releases so you should be able to just overwrite the files of the old version with the files of the new version. The only thing you'll need to do is either not overwrite<EMAIL_ADDRESS>or go to the skin's settings, right-click somewhere on the settings skin, and click on Rebuild main slots. Alternatively you can adjust any settings related to the slots (orientation, width, height, etc.), save, and close to force the main skin to be rebuilt according to your settings.

I copied the @resource folder onto the new one and I get this error again
NOTE (18:56:22.737) Lauhdutin.-.3.0.3\Main.ini: Refreshing skin
DBUG (18:56:22.762) : Initializing skin
DBUG (18:56:22.770) : Loading translation file
DBUG (18:56:22.781) : Initializing
DBUG (18:56:25.016) : Detecting games
DBUG (18:56:25.030) : Num platforms:    4
DBUG (18:56:25.037) : Registering platform processes
DBUG (18:56:25.044) : - 2 = Steam.exe
DBUG (18:56:25.051) : - 3 = Battle.net.exe
DBUG (18:56:25.057) : - 4 = GalaxyClient.exe
DBUG (18:56:25.064) :  Windows shortcut = true
DBUG (18:56:25.071) :  Steam = true
DBUG (18:56:25.077) :  Blizzard Battle.net = true
ERRO (18:56:25.084) Lauhdutin.-.3.0.3\Main.ini: Error: init.lua:10: The path to GOG Galaxy's ProgramData directory is not valid.

You can't copy the entire @Resource folder. You can copy @Resources\cache for the banners, @Resources\games.json for data relevant to your games (hours played, tags, etc.), and @Resources\settings.json for your skin settings from an older version.

That worked beautifully, had to take my shortcuts as well and then did the rebuild main slot and all good :)


Implement current CSAF schema

This issue tracks the progress to update Secvisogram to the current version of the CSAF schema f581451.

This has to include (at least) the changes from:

https://github.com/oasis-tcs/csaf/pull/228
https://github.com/oasis-tcs/csaf/pull/219
https://github.com/oasis-tcs/csaf/pull/218
https://github.com/oasis-tcs/csaf/pull/205


 ERESOLVE could not resolve
npm ERR!
npm ERR! While resolving<EMAIL_ADDRESS>npm ERR! Found<EMAIL_ADDRESS>npm ERR! node_modules/blockly
npm ERR!   dev<EMAIL_ADDRESS>from the root project
npm ERR!
npm ERR! Could not resolve dependency:
npm ERR! peer<EMAIL_ADDRESS>from<EMAIL_ADDRESS>npm ERR! node_modules/@blockly/field-colour
npm ERR!   dev<EMAIL_ADDRESS>from the root project
npm ERR!
npm ERR! Conflicting peer dependency<EMAIL_ADDRESS>npm ERR! node_modules/blockly
npm ERR!   peer<EMAIL_ADDRESS>from<EMAIL_ADDRESS>npm ERR!   node_modules/@blockly/field-colour
npm ERR!     dev<EMAIL_ADDRESS>from the root project
npm ERR!
npm ERR! Fix the upstream dependency conflict, or retry
npm ERR! this command with --force or --legacy-peer-deps
npm ERR! to accept an incorrect (and potentially broken) dependency resolution.
npm ERR!
npm ERR!
npm ERR! For a full report see:
npm ERR! /home/fenichel/.npm/_logs/2024-09-06T22_51_34_929Z-eresolve-report.txt

npm ERR! A complete log of this run can be found in: /home/fenichel/.npm/_logs/2024-09-06T22_51_34_929Z-debug-0.log

Some of these are the eslint issues that @cpcallen found before, and those can be resolved partially by removing the direct dependency on eslint and only using the dependency through blockly dev-tools.
The second set of errors are because field-colour wants<EMAIL_ADDRESS>as a peer dependency and neither 11.1.1 nor 11.2.0-beta.0 satisfy that.


[multistage] order-by on UNION STRING and INT columns somehow is allowed

SELECT val FROM (
  SELECT val, ord FROM (
SELECT CAST(1.1 AS DOUBLE) AS val, 1 AS ord
UNION ALL
SELECT CAST(3.1 AS DOUBLE) AS val, 3 AS ord
  )
  UNION ALL
  SELECT val, ord FROM (
SELECT '2.1' AS val, 2 AS ord
UNION ALL
SELECT '4.1' AS val, 4 AS ord
  )
)
ORDER BY ord

This query should throw an exception as 2.1 and 4.1 should not be allowed to compare directly against a numeric value (explicit cast is required) but somehow Pinot produces wrong result instead of throwing an exception.

related to #11878

Hi, I first raised a bug request for #11878, and I'm afraid this issue is wrongly reported. Since the order by is applied to the ord column, the strings are not compared to the numbers, so the query is correct.

@kim-jaewoong not really, if you change the order by to val it still works. b/c pinot considers literal as unparsed type until execution --> so it is forgiving enough to auto-cast 2.1/4.1 into double during comparison. i adjusted the issue description you can also try it out


Compile 3rd TypeScript modules with loose checking mode.

TypeScript Version:  2.8.1

Search Terms:
tsconfig + node_modules / compile third module
Tried solutions
set 'skipDefaultLibCheck', 'skipLibCheck', 'moduleResolution' fields in tsconfig.compilerOptions;
set 'exclude', 'include' fields in tsconfig;
Code
Given an example 3rd module viser(viserjs/viser) which are also written in TS. Its package.json defines types field guide tsc to reference 'src/index.ts' under its project root. The lib contains unused locals, unused parameters, implicit any (and we can even assume it contains some totally incorrect type assignments).
And the working project is written in strict mode, its compile configs have severely conflicts with the 3rd module which is written in loose mode. Are there any ways to compile them without error throwing?
Even though we can make tsc output compiled js files ignoring tsc errors, in create-react-app project, build by react-scripts-ts script, any compiling errors will be printed in the terminal and the dev page.
import viser from 'viser'  // a loose checking lib

// strict mode code below

Expected behavior:
The 3rd modules under node_modules are compiled in loose mode, not throw any errors. Our main
project is compiled in strict mode, throw conflicts with tsconfig rules.
Actual behavior:
Compiling through errors.

The 3rd party package is exposing its sources as output. that is wrong. it should be a .d.ts.
Exposing the sources of a package means you want every user to compiler and include that in their output. i.e. distributing code. in this case the errors are correct.
I suggest filing an issue against viser to use a .d.ts instead.


add support for UEFI HTTP Boot

Description
This adds supports for UEFI HTTP Boot.
Why is this needed
Fixes: #210
How Has This Been Tested?
Tested with:

QEMU OVMF (AMD64) (HTTP and TFTP Boot)
RPi4-UEFI (ARM64) (HTTP and TFTP Boot)
RPi4-UEFI-iPXE (ARM64) (HTTP and TFTP Boot)
Seeed Studio Odyssey X86J4105 board (AMD64) (HTTP Boot)
HP EliteDesk 800 35W G2 Desktop Mini (AMD64) (TFTP Boot)

How are existing users impacted? What migration steps/scripts do we need?
Should have no impact.
Checklist:
I have:

[ ] updated the documentation and/or roadmap (if required)
[ ] added unit or e2e tests
[ ] provided instructions on how to upgrade

/ok-to-test

@jacobweinstock, there were changes in the main branch that now conflict with this MR.
Its is safe to rebase over main? Or is there still some pending work that I should wait for?

@jacobweinstock, there were changes in the main branch that now conflict with this MR.
Its is safe to rebase over main? Or is there still some pending work that I should wait for?

Hey @rgl, should be safe. I don't foresee any big changes in the very immediate future.

@mmlb, @jacobweinstock, I think I've addressed all comments. Can you please review this again?

@mmlb, I think I've addressed all the comments (except adding the pxeClient boolean like commented above). Please check it out :-)

@mmlb, I think I've addressed all the comments (except adding the pxeClient boolean like commented above). Please check it out :-)

Yep, looks good to me. I'll just want to fire it up through a sandbox run to make sure all is good ;)

@mmlb with the latest push, the test should now pass.


In my opinion, if a linter crashes, gometalinter should:

mark this linter as failed (with an understandable message)
show stderr/stdout with flag --debug

I just recently ran into a similar problem. I hit a bug in gosimple that caused it to panic. But the panic wasn't noticeable from the output and it looked like linting was successful.  (I'll send a PR to update the gosimple vendor soon).
I think we should be able to detect panics. An exit code of 2 is a good sign. I'm not sure if any linters exit with 2 in other cases.
This would be a great addition. I should be able to work on a PR sometime soon if @alecthomas agrees.

I'm definitely in favour of correctly interpreting the exit state in case of a panic. However I'm pretty surprised an exit status of 2 is the result of a panic. IIUC that should not be the case. A SIGSEGV should result in an exit code of 139.
For example, a C program segfaulting results in this:
$ ./t
zsh: segmentation fault  ./t
$ echo $?
139

However an identical Go program does not:
$ ./tgo
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x205f]

goroutine 1 [running]:
panic(0x594a0, 0xc42000a230)
	/usr/local/Cellar/go/1.7.4_1/libexec/src/runtime/panic.go:500 +0x1a1
main.main()
	/Users/aat/Development/go/src/square/up/t.go:5 +0x1f
exit status 2
$ echo $?
2

This may be related to https://github.com/golang/go/issues/13484, though I'm not sure. There is a way to explicitly check for signal termination (outlined here), but if the Go runtime is suppressing the signal I'm not sure what can be done.
Digging deeper, it looks like the signal might be falling through. Odd. That may also not be the code resulting in the status 2, though.
Anyway, a solution would be good, but trusting a status of 2 is a panic is not sufficient. gometalinter itself, for example, can exit with a status of 2.

I'll send a PR to update the gosimple vendor soon

Note, I was hoping to remove that completely in #285 (if it gets the go ahead).

but trusting a status of 2 is a panic is not sufficient

That's what I was thinking. I think we would need to parse the stderr of the process to determine if the output looks like a panic.

Since the recent updates, the issue is now partially fixed: the full panic message is displayed.
DEBUG: warning: /home/pierre/go/bin/unparam returned exit status 2: panic: Package("runtime/cgo").Build(): unsatisfied import: Program.CreatePackage("unsafe") was not called

goroutine 1091 [running]:
golang.org/x/tools/go/ssa.(*Package).build(0xc4236406c0)
	/home/pierre/go/src/github.com/alecthomas/gometalinter/_linters/src/golang.org/x/tools/go/ssa/builder.go:2300 +0x1553
golang.org/x/tools/go/ssa.(*Package).(golang.org/x/tools/go/ssa.build)-fm()
	/home/pierre/go/src/github.com/alecthomas/gometalinter/_linters/src/golang.org/x/tools/go/ssa/builder.go:2260 +0x2a
sync.(*Once).Do(0xc4236406ec, 0xc4262e5798)
	/home/pierre/.gimme/versions/go1.9beta2.src/src/sync/once.go:44 +0xbe
golang.org/x/tools/go/ssa.(*Package).Build(0xc4236406c0)
	/home/pierre/go/src/github.com/alecthomas/gometalinter/_linters/src/golang.org/x/tools/go/ssa/builder.go:2260 +0x4c
golang.org/x/tools/go/ssa.(*Program).Build.func1(0xc42bf78440, 0xc4236406c0)
	/home/pierre/go/src/github.com/alecthomas/gometalinter/_linters/src/golang.org/x/tools/go/ssa/builder.go:2244 +0x2b
created by golang.org/x/tools/go/ssa.(*Program).Build
	/home/pierre/go/src/github.com/alecthomas/gometalinter/_linters/src/golang.org/x/tools/go/ssa/builder.go:2243 +0x11a


Create a project admin and mentor section in readme file

Is your feature request related to a problem? Please describe.
A section in the readme for project admin and mentors to recognize their hard work to grow this project.
there will be a photo (can get from GitHub) and ink their GitHub profile.

mentors are @Kajol-Kumari @Swarnimashukla @SKAUL05

@harsh-9in  would like to take up the issue

go ahead @shraddhavp :+1:


Lint YAML files

1. Issue, if available:
2. Description of changes:
Adds linting to yaml files.
YAML is a pretty easy format to mess up, and bad YAML files can cause weird unexpected results, hence it's important to protect the code base from introduction of bad yaml files and also lint our yaml files to enforce some level of standardization and cleanliness.
This adds a yaml linter to the repo, for now it disables checking for all the problems that we already have, we can later on enable them one by one after fixing the issues.
3. How was this change tested?
4. Does this change impact docs?

[ ] Yes, PR includes docs updates
[ ] Yes, issue opened: link to issue
[x] No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

I'd love to include this in make verify rather than in a GH action. It's not easy to run GH action code locally, and ideally this would be executed as part of the dev workflow. In general, I lean away from GH actions that do more than trigger scripts that we can run locally.
Bonus points if such yaml linter can be installed with go install. Unfortunately, I'm not finding a good one. There are some benefits to GH actions, such as not needing a cross-platform install story locally. Go based tools are easy, but python/js are nastier to include in the dev loop. Let me know if you find anything, otherwise I'm happy with GH action approach.

Still working on this.


Overhaul pre-processor to include shapely

Is your feature request related to a problem? Please describe.
No specific problems, but would increase functionality and simplify a fair bit of the code.
Describe the solution you'd like
Use shapely in all it's glory.
Describe alternatives you've considered
Writing the code yourself - tried that, no fun.
Additional context
Suggesting in issue #9 by Agent6-6-6.

Hi,
Am sure I saw a 'DWG import' todo somewhere - this is more related to that, but I am using shapely.
A WIP but (as part of another project) I made this: https://github.com/aegis1980/cad_to_shapely. I am having a few issues with making is work on PyPi (imports mostly: Python not my goto) but at some point it will be a pip-able thing.
Currently only DXF (I ve written code for IGES and SVG - need to add it).
Anyway, this gist imports a (Rhino3D generated DXF) aluminium extrusion (file attached) and spins it around for sectionproperties processing.
Need to check I get the same answers as I get using Strand7 so unverified.
Good work BTW ;) And a strong antipodean influence too.

section_holes_complex.zip

Hi @robbievanleeuwen,
I have some time before January and I would like to work on this. I am experienced with shapely and have previously started looking into how to integrate it into section-properties.
I will make a fork and start working on a branch so you can review the progress. As I go, let me know if you would like to advise me further or chat about the direction it's going.
Thanks for this excellent library! I think using shapely will be an excellent addition to it.

Hi @connorferster,
I initially had a few tries at implementing shapely but ran into a few hurdles and didn't delve into the docs deep enough to pull through. If you think you could make it work I think it would be a fantastic addition and potentially open up a few other possibilities with sectionproperties, so I'm more than happy for you to give it a try!
Give me a shout if you have any questions or would like to discuss anything 👍
Cheers,
Robbie

Now, I am able to get test_rectangle.py to consistently run in pytest and I no longer have my issues when creating the mesh in Jupyter when also using matplotlib to .plot_geometry(). Great news.
Additionally, my tests were crashing because of PlasticSection: I had not reviewed it or updated it yet to see if there were inconsistencies with the new pre-processor. However, my intention is to fit the new pre-processor into the existing CrossSection and analysis interface so it should not require updating.
I am however getting some funny values when calculating some geometric properties such as Ixx_c/Iyy_c and qx/qy. Looking into it now....

Now, I am able to get test_rectangle.py to consistently run in pytest and I no longer have my issues when creating the mesh in Jupyter when also using matplotlib to .plot_geometry(). Great news.
Additionally, my tests were crashing because of PlasticSection: I had not reviewed it or updated it yet to see if there were inconsistencies with the new pre-processor. However, my intention is to fit the new pre-processor into the existing CrossSection and analysis interface so it should not require updating.
I am however getting some funny values when calculating some geometric properties such as Ixx_c/Iyy_c and qx/qy. Looking into it now....

The plastic section calculation involves regenerating geometry and remeshing so I'm not surpirsed this isn't working yet! Give me a shout if you need some help understanding how it works 😄

Thanks @robbievanleeuwen, will let you know as I get to it (next week?)
I found what was failing my tests with the qx, qy, Ixx_c, Iyy_c, etc. values. It seems that the reported values are actually Eqx, EIxx_c, etc. and not yet pure geometric value. It seems that these are values reported from the MeshPy elements and that sectionproperties would have to divide these values by E. I was running the tests with a "steel" material of E=200e3 and would not have expected the E value to affect the geometric properties. I realized this when I ran the display_values function and clued in that you do actually report them as E.qx, etc. Once I changed to E=1, I started getting exact match values.
Are you open to altering this behaviour so that the geometric properties are reported as pure geometric properties while the current reported properties being relabelled as something like Eqx?

The current behaviour switches to outputting 'material affected' properties as soon as a material property is specified. So if you wanted a purely geometric analysis with one material, there would be no need to specify the material.
I get that this is a little confusing and the reference to this behaviour is buried in the docs (probably not in the most obvious section). I think you've probably highlighted that this is not really a very good way of going about things and can cause a lot of confusion and wasted time (sorry!).
Definitely open to altering the behaviour, noting that as soon as you have more than one material a lot of the section propreties become undefined and therefore need to be modulus weighted.

Aaah...I see it now. I take it back, I think your existing approach is best. I was thinking of the "default material" being a bit of an after-thought just because I am so used to having to specify a material to do any analysis in commercial software. However, it's a good solution: a gift to the user that I was foolishly rejecting!

I think it probably needs a bit more of a highlight in the docs - opened issue #53.

Was modelling a funny-shaped hollow column section today in some commercial software. It was so painful. I wished I could have been using the new preprocessor functions!!! Would have been so much easier! I really want to get this live...

Hi @robbievanleeuwen,
I am noticing some funny behaviour with the geometric properties. They seem to be dependent upon the section's absolute location within the cartesian plane. For example, using the rectangular_section() function to create a box centered on the origin, I generate the following results:

Note: qx, qy, specifically.
If I shift the section so that the origin is the bottom left corner, I generate the following results (which would be considered the "correct" results):

If I shift the section further still:

Note: Ixx_g, Iyy_g, and Ixy_g also get out of whack.
Have you encountered this behaviour before? Is this expected and I just missed it in the docs?

Hi @connorferster this is expected behaviour. Ixx_g and qx are taken about the global axis, which is (0,0). So these values should change as you move the location of the section around the cartesian plane. I would expect qx=qy=0 when the global axis coincides with the centroid in example 1. Ixx_c is taken about the centroidal axis so remains unchanged in the various examples. I think the various axes are described in the docs, e.g. here.

It seems I changed the implementation of rectangular_section() (at some point...for some reason...?) to return a rectangle centered at the origin, instead of your original implementation centered on b/2, d/2. No idea why I did that, especially since I want to keep all of your shape generating procedures the same as you made them. This explains why the rectangular_section() results were inconsistent...solved!

Hi @robbievanleeuwen,
I am having trouble with the plastic properties: some of the values with the shapely pre-processor ("shapely") are incorrect when compared with the main branch ("main"). I am investigating it by using the same CircularSection used in the examples. Here are some of the things I have found so far:

Point coordinates and facets are identical between "shapely" and "main"
Number of mesh elements are the same between "shapely" and "main"
Plastic properties for each element are identical between "shapely" and "main" for some elements but different for other elements
Number of iterations to converge with the brent root finding algorithm (via CrossSection.pc_algorithm) is just over double (four iterations in "main", 8-10 iterations in "shapely")

My plan is to next to check is to if every Tri6 element is identical. However, I am wondering if these symptoms are familiar to you during your development of the library.

Hi @robbievanleeuwen,
Further to the chain of information above:
Edit 6:
The Section/CrossSection mesh nodes and elements and attributes are all identical to the 16th digit at time of instantiation. Additionally, the Tri6 elements between "main" and "shapely" are also identical to the 16th digit.
Edit 7:
Once the Section/CrossSection is instantiated, the next step I take is to do .calculate_geometric_properties. Almost all of the individual calculated values in each element (e.g. area, qx, qy, etc) is different at either the 12th or 13th decimal place. The occasional element has a matching value but they are mostly different with, what appears to be, the same inputs. I thought that the problem may have been the incremental sum but each individual element properties are slightly different.
I am now stumped.

Shapely log file:
https://drive.google.com/file/d/1c2tlHwUAV7uZeiXkZkRFBS9Gp3BhTRjo/view?usp=sharing
"Main" log file ("legacy"):
https://drive.google.com/file/d/16MiJFBQHEJwQc_FboirErJYlkFGZrqGw/view?usp=sharing

@connorferster, are you possibly just running into numerical precision issues here?
While the numbers differ, they are still pretty darn close to zero!
I wonder if you're taking more iterations (presumably using Brent's solver in scipy.optimize.brent), then there's something to look at there. Because it seems odd that this would be the case for what is the exact same problem as you describe it.
Does the same occur with other section types?
I must admit I've only partly kept up with this thread and not taken the shapely version for a spin yet, nor looked at the code specifically.
The only time I've seen Brent algorithm spit out slightly different answers (and also generally take more iterations) is if you set the tolerance to large, for example having for example 5 decimal places of precision might take more iterations than 8 decimal places. It sounds counter intuitive but going smaller sometimes results in less iterations, it's probably something to do with how many times you're evoking each type of interpolation in the algorithm or some other weirdness.
I'm sure you've checked you're using exactly the same input parameters for scipy.optimize.brent.

Definitely running into floating point precision errors here :)
What is really stumping me is the question of where the error originates? I am using Robbie’s exact same code to generate point coordinates with the only real difference being they first go into a shapely object before going into Geometry.points. So is something happening within the shapely object somewhere beyond the 16 decimal that is preventing me from getting matching results?
That is my thought for my next test. But, gawrsh...
Sent from my iPhone

On Feb 8, 2021, at 23:30, Agent6-6-6<EMAIL_ADDRESS>wrote:

@connorferster, are you possibly just running into numerical precision issues here?
While the numbers differ, they are still pretty darn close to zero!
I wonder if you're taking more iterations (presumably using Brent's solver in scipy.optimize.brent), then there's something to look at there. Because it seems odd that this would be the case for what is the exact same problem as you describe it.
Does the same occur with other section types?
I must admit I've only partly kept up with this thread and not taken the shapely version for a spin yet, nor looked at the code specifically.
The only time I've seen Brent algorithm spit out slightly different answers (and also generally take more iterations) is if you set the tolerance to large, for example having for example 5 decimal places of precision might take more iterations than 8 decimal places. It sounds counter intuitive but going smaller sometimes results in less iterations, it's probably something to do with how many times you're evoking each type of interpolation in the algorithm or some other weirdness.
I'm sure you've checked you're using exactly the same input parameters for scipy.optimize.brent.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

I am having trouble with the plastic properties: some of the values with the shapely pre-processor ("shapely") are incorrect when compared with the main branch ("main").

How different, in the end, are your results compared to current master ("main")?

This may be a silly question, but does the exact same geometry input (in master) always generate the exact same output to those 14th-16th decimal places? It may be some chaos introduced with the optimization algorithm. Since the differences are so small, I would tend to note the difference, and ignore them for now; relying more on an overall test for similarity.
Yes, it's very curious that they're different, since the shapely object should be exactly the same input. Is the modified code with your logging on your branch already?
The fea.shape_function function doesn't seem to be doing anything that would cause J to differ. It's all based on the gauss points, which seem to be hard-coded anyway. So I'm not sure where the deviation on the Jacobian is even coming from. The only difference is the coords  np.array passed to shape_function. But you confirmed already that the points are exact out to 16 digits, right? So coords should match exactly between master and shapely-branch. 🤔 😕

@Spectre5

How different, in the end, are your results compared to current master ("main")?

For the geometric properties: a very small fraction of a percent with discrepancies occurring in the 6th to 8th decimal places
For the plastic properties: the results are just wrong.
It was observing the incorrect results (and incorrect intermediary values) that started me on this chase. I noticed the slight discrepancy in the geometric properties earlier on but, because they were so small, they did not particularly bother me. However, once I noticed that the plastic properties were completely incorrect, I started wondering if there was a relationship between these two things.
@Czarified

This may be a silly question, but does the exact same geometry input (in master) always generate the exact same output to those 14th-16th decimal places?

I don't think that's silly at all! That was something I checked early on, too. All of my runs on the "main" branch with the circular section geometry exactly matches the results published in the first example in the Docs. So, I scratched that possibility out.

Is the modified code with your logging on your branch already?

Just pushed it. You can find it on my repo under the "shapely..." branch

Hi all,
Unpacking the above, there seems to be two issues:

Precision errors - I don't feel like I have a deep enough understanding of computer science to be able to help on this one. And perhaps this is my naivety/lack of compsci knowledge, but are we particularly concerned about variations in the 12th decimal place? My guess would be that there are some extra operations within shapely that mean precision comes into the equation but that's just an uneducated hunch.

Plastic properties - I wouldn't expect the algorithm to take twice as long to converge. There are some usfel debugging tools built into the plastic section analysis which might help you get to the bottom of this. They visualise each step of the algorithm and show the trial plastic neutral axis geometry and the generated mesh as well as print some useful info to the terminal. I would try setting verbose=True and debug=True - docs here. This may help you compare what the plastic algorithm is doing in each step for the main branch and the shapely branch.

@Czarified to answer your question: plastic analysis only computes areas - when we compute areas 3 point integration gives exact results (in fact one point integration would be enough as the Jacobian is constant in the Tri6 element!) The geometric analysis requires higher order integration as we are integrating higher order functions which vary over the Tri6 element, e.g. the computation of second moments of area has quadratic and mixed terms - this needs six point integration to get an exact result. I talk about this briefly in section 4 of this.

I'm not concerned about the 12th decimal place being inaccurate. If there are minor differences and it's decided that 12 significant figures is where it's at in terms of accuracy, then just state that in the docs and you could even give an option to hard round to that many significant figures if desired so answers are always the same. The fact that you have to define curved regions as a discrete number of straight facets kind of throws even achieving a few decimal places of accuracy out the window anyway for practical uses.
Perhaps some of the plastic properties can be worked out using shapely, taking the section perimeter and splitting it using a horizontal line. Work out plastic modulus from centroid of each divided half once you have the dividing line in the right location to make area each side equal (or if you have different yield stresses use the values weighted by yield stress). I say this as an alternative method to just to give you another result to see if it is the same or not as one of the results to understand if there is an issue. You don't need to mesh the full section necessarily to determine the plastic axis and plastic section modulus unless I'm mistaken.

@robbievanleeuwen Thanks&kudos for link to your paper (plus I checked out your other posts, inc Samsung stairs). Nice. Good to see Strand7 still a goto in both!
@'everyone' (not a thing) Got some time next week so going to run 'my' import dxf code (see previous) against new shapely implementation. Will try and keep you updated.

Hi @robbievanleeuwen, @Agent6-6-6, @Spectre5, @Czarified,
First, thank you for your support and suggestions. I really appreciate it!
Update: I believe I found the problem with the plastic section properties. I had had a previous exchange with Robbie where I noticed that there were methods in Geometry for adding individual lines and points to the section and I had taken them out after we both thought that would no longer be necessary with a shapely interface. As such, I began commenting out lines that referenced these methods (or lines that I thought were referencing those methods in CrossSection). As such, I broke a critical piece of the plastic section calculation where the section is cut along the principal axes.
However, I think that shapely can have an easier time of this with shapely.ops.split(geometry, splitter) and I can shorten the code for PlasticSection.add_line.
Going through this preprocessor update, I am really appreciating the amount of work Robbie did to build a geometry engine from scratch. It's excellent.

Hi @connorferster,
Sorry I remember this exchange and it seems like I gave you some dodgy advice - looks like I did use these methods after all!
I think that both your suggestion of using the split method in shapely and @Agent6-6-6's suggestion of not requiring meshing are great! The only reason I need to remesh in the plastic algorithm is to recompute the area above and below the trial plastic neutral axis - however if there is a way to get this info from shapely we can remove the constant remeshing from this algorithm completely and will simplify the whole thing!
Thanks again :)

Great investigative work!  I like the idea of potentially looking into other solutions for getting area "above" and "below".  It would probably be faster too instead of meshing and re-meshing many times.

@robbievanleeuwen
I was scratching my head for the last couple days wondering why my warping properties tests were not passing the test suite where yours did! Then I noticed that you did not actually test the warping properties ;) When I actually connected the test for warping properties in test_rectangular.py then the results from the main branch failed the tests also.
When I checked my results against the main branch results, they were the same but neither (completely) match the analytical results from the test suite (within the prescribed tolerance, anyway).
I will change the test suite expected results to match the results calculated from the main branch, taking the main branch results as correct (for now, anyway).

Hi @robbievanleeuwen,
The original test suite now runs and passes, which is great. However, I wanted to a create a section that is a more elaborate test but the darned thing keeps crashing in the last iterations of the plastic section calculation. It is crashing Triangle. Here is the error message I can generate when I run a my test script from the console:
Internal error in segmentintersection():
  Topological inconsistency after splitting a segment.
  Please report this bug to<EMAIL_ADDRESS>  Include the message above, your input data set, and the exact
    command line you used to run Triangle.

Here is a gist of the same script to demonstrate what I am doing:
https://gist.github.com/connorferster/46d46881b10e844a8bd90ed926ec7d58
It seems that some funny geometry is being created which is crashing Triangle. However, when I use the debug=True flag in .calculate_plastic_properties() I cannot discern where the broken geometry is.
I have gone line-by-line through the small amount of modifications I have made to cross_section.py, fea.py, and pre.py to try and find something that is breaking it. Of course, when I output the points, facets, holes, and ctrl points from my "compiled" geometry and feed it into the main branch as a CustomSection, it works. I have been doing line-by-line comparisons of the main branch to try and find what the problem is and I cannot find it.
Would you be willing to do a web conference chat sometime in the next week to review together? I am hoping that you may be able to see something that I cannot see.
I am so close to getting this closed out and a pull request submitted. Hoping that we can get this fixed together.
Many thanks,
Connor

I can understand wanting to get to the bottom of this issue, even just to be sure there isn't some deeper or bigger bug.  I know I'd want to.  But that said, once figured out, are you still thinking about looking into switching to shapely to do this plastic computation part instead of using triangle to continually remesh it?  Or do you feel that is better saved for a separate PR?

I think it is a separate PR. I started looking into it over the last week and I realized that there would be significant code changes to cross_section.py to make it work. It is well beyond the scope of what I want to contribute on this PR (which itself is a pretty big scope!). While I have previously been playing around with the mechanics in shapely that can perform this calculation, I have not gotten the process 100% clear in my mind yet.
The shapely preprocessor is connected so that the data coming in to the analysis modules should appear no different than if it were coming from the Geometry objects on the main branch. I fixed the previous plastic  sections problem by removing a “modification” (aka naively breaking the code) I made to cross_section.py. So I am surprised now that this new issue has occurred.

On Feb 15, 2021, at 22:53, Spectre5<EMAIL_ADDRESS>wrote:

I can understand wanting to get to the bottom of this issue, even just to be sure there isn't some deeper or bigger bug. I know I'd want to. But that said, once figured out, are you still thinking about looking into switching to shapely to do this plastic computation part instead of using triangle to continually remesh it? Or do you feel that is better saved for a separate PR?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

Makes sense.  Have you tried rounding the points location that Triangle gets?  It might at least tell you if the issue is another numerical precision issue

Hi @connorferster, sorry I've been MIA the last 2 weeks. I'm back at uni (again!) and it's just kicked off so I've been a bit preoccupied!
Good find with the testing, it seems like I never finished that file - very strange!!
Very happy to have a chat via a video call, would be nice to meet as well! Maybe the easiest way to organise would be to flick me an email at<EMAIL_ADDRESS>and we can find a time that suits both our time zones ;)

@robbievanleeuwen @aegis1980 @Czarified @Spectre5 @Agent6-6-6
The new shapely pre-processor is finished and 100% operational. I have just made a push to my fork and have posted this (lengthy) gist to showcase most of the new features:
https://gist.github.com/connorferster/29a93ab1d07741d9ab4306a8e458a083
As I understand, Robbie is quite busy on a new venture right now. A community has been gathering around sectionproperties which I think is really special. I would really appreciate for all interested parties to review the above gist and to review the code in the new sectionproperties.pre.sections file, where the lion's share of the pre-processor resides. How do you feel about the code? Is it reasonably easy to understand? Do you think it looks maintainable?
Additionally, if people have time and interest, please try cloning my fork onto your machine and install it with setup.py and try it out for yourself!
At the end of the gist, are three action items I would love, love, love to have some help with. Particularly updating the documentation. Please drop me a line if you are interested in assisting with updating examples in the documentation with me. If there are more of us pitching in, we could probably get it done in a short weekend session!
Thanks for everyone's support and interest in this new pre-processor! I hope that we can continue building sectionproperties for years to come so that there is a free software option for this fundamental piece of mechanical and structural engineering.

@connorferster, finally taking this for a spin to test....
I thought it would be useful to start a bit of a list on things that are different to enable people to convert existing scripts once things get merged, working through converting some older scripts to the shapely version I've noted the following so far:-

CustomSection no longer exists, similar functionality is provided now by .Geometry.from_points
All the previous built-in geometry definitions have been renamed slightly (typically the 'shift' parameter has also been removed). Can still shift using shift_section after defining geometry.
It might be useful to have a before/after table correlating the above name changes in the documentation to help people mix and match the changes
mirror_section, shift_section, etc works differently than previous, you can no longer apply directly to a created geometry, before you could go geo1.mirror_section(….), now you have to go geo1=geo1.mirror_section(….) (as far as I can tell if I'm interpreting things correctly?)
Mirror_section works differently (see point below as have not quite figured it out....)
shift_section works differently, now accepting two separate variables for x and y offset instead of a list. Can no longer be applied like noted above for mirror_section example directly to a geometry instance
MergedGeometry is now renamed CompoundGeometry

Couple of other points as I progress through converting a single script that I've noted:-

I had to manually change the shapely.log path in sections.py as it was hardcoded to your desktop path (probably not intended for final code obviously)
I required to install icecream package, this isn't a requirement in the setup? Include in requirements if intended for final code?
I required to update my matplotlib (had very old version (like 3.1.2)) that apparently didn't have matplotlib.colors which is required somewhere, might need to enforce some minimum version here?

This is probably me being dumb, but the example given in the docs for mirror_section doesn't provide an example of mirroring about an arbitrary point, and since things have changed I'm not entirely sure how to get it to work.
Previously this following worked geometry5.mirror_section(axis='x', mirror_point=[0, 0])
Looking at the docs it has Union[list[float, float], str] as the type, what is the str if I specify a specific point? I'm missing something but not sure what?
I can do geometry5 = geometry5.mirror_section(axis='x', mirror_point=[0, 0, 0]) and it gets past the line but simply trying anything to try get things to work ... :), not sure what I'm supposed to put in the position of the third zero? Throws an error if you only have [x, y] coordinates as implied in the docs?
TypeError: can only concatenate list (not "tuple") to list
Can you provide a small example of how mirroring might work about an arbitrary point as I'm clearly missing something (probably obvious)! Thanks

@connorferster
One other thing I just noted, in plot_mesh, materials=True is the default. This puts the label on the plot, but if you have no materials defined I'd probably prefer this to be set to false as the default option. Presumably you can check if there are materials defined, and if none are defined set this argument to False? This was the previous behaviour, a plot of the mesh produced with no legend.
Theres also a debugging print(legend_list) that is probably left over in plot_mesh?
https://github.com/connorferster/section-properties/blob/77cd4a8b36a4d335b41a74f70a3a894d8771f68b/sectionproperties/analysis/cross_section.py#L1072
One other possible improvement:-
Is it possible when you have multiple geometry elements and if one mesh_size is specified with create_mesh to just to apply to all elements irrespective of number as a default behaviour?
Currently you need a list with the same number of mesh_size entries as geometry elements. But I find I'm usually just applying the same mesh size throughout.

@Agent6-6-6
Agreed! A comparative list would be a good plan.
You have certainly caught some of old instrumentation there (logs, icecream). Thanks for catching those. Will remove.
More tomorrow...

Hi @Agent6-6-6
Thanks for pointing out those gaffs. I have fixed .mirror_section() and removed the instrumentation and print statements.
Regarding a list of old vs new, do you think it would suffice to redo all of the current examples in the documentation with the new preprocessor, showing them side-by-side? Or with old code commented out with new code below each line?
What do you think the best format may be for presenting the comparison?
Thanks!

@connorferster, I was imagining just a table or list describing the differences as a separate page in the docs would suffice. It's really just to give users a bit of a headstart in making the required adjustments to their previous scripts. But an example that uses/demonstrates the majority of the changes to old methods and demonstrates the different approaches/changes compared side by side has merit with some appropriate code comments explaining the differences. I don't think you need to explain any new features here, just what might need to change in older code to update for shapely pre-processor.
I'm not sure how @robbievanleeuwen intended to treat the new shapely version, as next version 1.09 or whatever its up to, or a completely new package? That might have some bearing on how the docs are approached. I'd imagine it's just a natural progression to next version within the same package. So all the old methods become redundant and if someone wants to find out about them they can install an older version or review the older versions of the docs.
I don't really see any benefit in providing both versions in the docs (not withstanding it would be a significant amount of work to do both I'd imagine). If someone wants to learn about the previous code nomenclature they can drill back to an earlier version of the docs anyway.
I'll let you know if I note anything else if you've fixed mirror_section 👍

Thank you @Agent6-6-6 for reviewing the code. I really appreciate it!
That sounds good. I will play around with a couple of ideas to see what is perhaps more communicative and comprehensible. The documentation will need to be updated for the new version anyway. I can add a page of "API changes from v1.x.x" or something to that affect.
Because this will be introducing several breaking changes to the code, I think the v2.0.0 will be most appropriate (if following semver, which I think is a good idea).

Hi @connorferster, I've only just got back onto testing your latest changes and I note in fea.py there is the same hardcoded log path that was in sections.py at the start. Probably needs some tidying up as well as currently errors on the following code:-
https://github.com/connorferster/section-properties/blob/3266c21313d63673b6b3c9c8048bb5fec7fdfb2c/sectionproperties/analysis/fea.py#L11

@connorferster, the following lines put into create_mesh routine after     mesh.set_holes(holes)  # set holes line will achieve this request:-

One other possible improvement:-
Is it possible when you have multiple geometry elements and if one mesh_size is specified with create_mesh to just to apply to all elements irrespective of number as a default behaviour?
Currently you need a list with the same number of mesh_size entries as geometry elements. But I find I'm usually just applying the same mesh size throughout.

    if len(mesh_sizes) == 1: # if single mesh size value provided, create meshes for all regions at this size
        mesh_sizes = mesh_sizes * len(control_points)

@connorferster , at this point, do you want to submit it as a PR, so we can give a more "formal" review? GitHub has built-in functionality for this, and it will be documented in the same PR that implements the changes, for record-keeping. I'm going to take an initial look today, but I don't want to build/install the branch on my work computer right now. I'll dig in deeper at home.

Yeah will do.

Confirmed! Will do this eve.

PR Submitted

Hi @connorferster
This just tripped me up as another subtle change, argument mesh_sizes has changed to mesh_size in the Geometry Class create_mesh function.
This is an issue because if you have code that could result in either a single or multiple geometry element the expected/required kwarg seems to change... for example adding or not adding a strengthening plate to a I-section based on some code logic requires different arguments.
Then if you pass a single geometry element it fails using mesh_sizes as an argument, but if you pass a compound geometry it works using mesh_sizes as an argument. Conversely it will work with a single geometry element using mesh_size as an argument, but fail if there is compound geometry using mesh_size
If that made no sense I can write an example to demonstrate. But I believe it should universally be one or the other here if there is no reason to change to mesh_size, then keep as it was?
Either have it one way or the other
(appreciate the documents not fully updated, but in most locations it would need updating to reflect the final decision)

@connorferster , also the external perimeter output seems to be incorrect for compound geometry. I think it's returning the bounding box perimeter rather than the true external perimeter of a section (see example below). Still seeing if I can figure out why in the code, but you might know why if more familiar with shapely and how it's implemented.
For example:-
perimeter = 1788mm (section tables put 310UC97 as 1790mm)

compared with adding 2 330x16 side plates, external perimeter should be 1378mm but the code is returning 1334mm which is the perimeter of a bounding box

what it seems to be reporting back is this, vs the true perimeter:-

Compared to v1.08 I don't think the perimeter output worked at all for merged geometry, just reporting zero.
Seeing almost double the speed executing code which is really nice 👍

Regarding the perimeter, finally found it in the code, it's because section-properties is using the convex_hull function from shapely. What we require in this situation is the concave hull of the compound geometry.
Unfortunately, shapely does not include a concave hull formulation. The problem being there is no single answer to the problem, it's really dependant on the concave hull algorithm you're using.
I found this which may be of some use. https://gist.github.com/dwyerk/10561690
Will do some more reading....

@Agent6-6-6
Thanks for the reminder on the compound perimeter. It is something that I became aware of a couple months ago and, at the time, did not know how to make work. Now I think I can get a correct value for a compound geometry. Should only take an extra line or two if code now.
Regarding the API change of mesh_size (Geometry) and mesh_sizes (CompoundGeometry), I made that change about a week ago thinking that would be more descriptive and serve as a reminder to the user of the distinction between Geometry vs CompoundGeometry. Reading your comments, however, I am struck by how troublesome it is to have two different argument names. It feels like an obvious choice now to keep the same argument name for both.
I think I would rather make the arguments obvious and consistent vs writing documentation to explain it!

@Czarified Re: perimeter. I have not used it yet myself. I just think of it as "nice to have" info in case someone ever does need it. Now that it has occurred to me how to do it, we won't have to write the extra documentation! XD

Geometry.create_mesh() now as arg 'mesh_sizes' instead of 'mesh_size'.
CompoundGeometry.calculate_perimeter() now calculates exterior perimeter if the geometry is not disjoint. If disjoint, method returns -1.

@Czarified , @connorferster , the reason I originally requested the perimeter was added was for two reasons

required in Australiasia for fire calculations to calculate the heated perimeter.
required for determining coatings coverage (allows assessment/comparison of various options)

If its possible to return in shapely a merged geometry and fill the holes ignoring any internal holes (get geometry of holes and add as a union as external regions instead?), then you can use boundary.length to get external perimeter. I started going down this route last night but not being so familiar with the shapely and how you've used in for the geometry I wasn't having much success.

@connorferster, one other subtle change I just noted trying to track down why the time_info = false default wasn't working for me with the Section class that's worth adding to the list of differences for converting scripts:-

As mentioned previously CrossSection class changed to Section class, with the change to shapely, as I understand it the mesh and materials are now part of the geometry object, whereas previously they were not. I hadn't picked up on this previously as was inadvertantly feeding the mesh in place of time_info as hadn't realised the arguments changed and wasn't using mesh=.... which would have thrown an error obviously!

So previously you had to feed the mesh & materials into CrossSection class as arguments. Now you don't need to with the new Section class (much simplier by the way).

Previously section = CrossSection(geometry, mesh, materials, time_info) for example
Now section = Section(geometry, time_info) only for example

I guess if any other class arguments have changed arguments its worth summarising the differences in this list as well.

Hi @connorferster, seeing some weird behaviour with the following example if you wouldn't mind having a look.
I'm pretty sure this should be working, but I see the shear centre being way off the section when it should lie on an axis of symmetry given the arrangement (it worked fine in v1.08).

The code is part of a larger example with a few more geometry elements for welds joining the two elements that I just couldn't get the shapely version to create a mesh for (just did nothing hanging on create_mesh command), so I started removing geometry until I saw the shear centre being out here in this example and thought this might be the reason why my larger example wasn't working as something isn't maybe meshing properly. Could you take a look when you have some time as definitely something weird going on.
import sectionproperties.pre.sections as sections
from sectionproperties.analysis.cross_section import Section

d = 612  # I section depth
b_f = 229  # I-section flange width
t_f = 19.6  # I-section flange thickness
t_w = 11.9  # I-section web thickness
r_1 = 14  # I-section root radii

d_tee = 202  # strengthening tee depth
b_tee = 178  # strengthening tee flange width
t_f_tee = 12.8  # strenghtening tee flange thickness
t_w_tee = 7.8  # strengthening tee web thickness
r_1_tee = 11.4  # strnegthening tee root radii

n_r = 5  # number of points considered around root radii

mesh_area = 20  # max mesh size

# I-Section beam
geometry1 = sections.i_section(d=d, b=b_f, t_f=t_f, t_w=t_w, r=r_1, n_r=n_r).align_center()

# strengthening tee
geometry2 = (
    sections.tee_section(d=d_tee, b=b_tee, t_f=t_f_tee, t_w=t_w_tee, r=r_1_tee, n_r=n_r)
    .mirror_section(axis='x', mirror_point=[0, 0])
    .align_center()
    .align_to(geometry1, 'top')
)

# total number of individual geometry elements
geo_number = 2

# assemble geometry list
geo_list = [globals()[f'geometry{i}'] for i in range(1, geo_number + 1)]

# create merged section
geometry = sections.CompoundGeometry(geo_list)

# plot geometry
geometry.plot_geometry()

# create mesh
geometry.create_mesh(mesh_sizes=[mesh_area])
# create section
section = Section(geometry=geometry, time_info=True)

# calculate results
section.calculate_geometric_properties()
section.calculate_plastic_properties()
section.calculate_warping_properties()

# plot results
section.plot_mesh()
section.plot_centroids()

# display all results
section.display_results(fmt='.3f')

@connorferster  if you comment out the line .mirror_section(axis='x', mirror_point=[0, 0]) in the tee_section code you'll hopefully see the issue of the mesh not being created.
By the way, really liking the .align_to and .align_center options. Makes it really easy to create compound sections vs how it was done previously with the .shift_section being the only option 😄

Hi @Agent6-6-6,
That is really interesting. Naturally I wanted to check this out right away!
When I tried doing the same, I did not get the erroneous result. Now, I would like to reproduce the result you got so I will do a straight copy-paste of your whole script next.
I tried to upload this as a Gist but Github was giving me an error so apologies for the screenshot:

Yeah, copy-pasted your whole script and got the same issue you did. Weird...

Oh, interesting. The problem seems to occur when the geometry is centered on the origin.
In my first script, I missed how you aligned the I-section to the origin, so my script did not work as-is. I had to alter it so that the tee aligned to the center of the I-section and it worked fine.
When I made that one alteration to my script, aligning both the I-section and tee to the origin, then the shear center wigged out. I didn't have a problem with meshing though. Did you have a problem with meshing with your script as you have posted it?

@connorferster , if you take the script I posted and just comment out the mirroring of the tee I see it failing or getting stuck on the mesh creation. Do you get the same behaviour?

@Agent6-6-6 I do see what you are talking about. The mesh creation seems to "hang".
Interestingly, I don't have the hang problem when the section is not centered on the origin. This centering on the origin seems to be the key to this strange behaviour...

@Agent6-6-6 I have been playing with this for about an hour now and it is so strange. It seems to be something about this particular set of points and facets. Question, if you were to print geometry.points, geometry.facets, geometry.control_points, and geometry.holes and then use those values in v1.0.8, do you get the same hanging behaviour? This is my first time having a "hang" from the meshing. Typically I just get something like a segment fault that crashes my Python kernel.
More weirdness about this:

I put in a bunch of "instrumentation" (i.e. print(...) statements) going from right before meshpy is called in pre.pre.create_mesh() and almost every step back to the original two methods, Geometry.create_mesh() and CompoundGeometry.create_mesh() where I put a print() statement as the first line in both method calls. When I go to call geometry.create_mesh(), I see nothing. No prints, not anything.
If I then change the approach to building the CompoundGeometry as the way I did in my above screenshot then no problems. I see all of the instrumentation print out as expected and it meshes quickly.
This script you have has the same behaviour consistently.

Given that I cannot see any of my print() statements execute, even from the very beginning of the .create_mesh() method call. I am stumped, I feel like I cannot track down what the problem is if I don't know what code is running and what code is not running! I am at a bit of a loss on this one...Just did a quick search and found some options for debugging this kind of thing on stackoverflow. Will try tomorrow.
In the meantime, I found out why your shear center is coming out funny on the mirrored section. It's the way you are aligning the objects: they are not touching.
When you use align_center() in this manner, where both geometries are aligned to their respective centroids independently and then aligned with align_to(...):

You can see that the resulting SVG is green. This means it's a valid plotly shape because the two shapes are either only touching at a single point or they are not touching.
When you use align_center(geometry1), then you are aligning one geom center to the other geom center and now the SVG is red indicating they are touching at more than one point (i.e., they share a line or are overlapping, which is what we want):

So, why does this happen? Good question, especially in this case because you are using align_to(geometry1, 'top') and I would have guessed that that would do it. I am guessing it is due to teensy floating point differences. Am interested to see why .align_to(geometry1, 'top') is not fixing it, though.

@Agent6-6-6
More info:

I tried using hanging threads to see if the fault could be found but no fault was triggered
On Windows, I pulled up the Task Manager while it was running and Python was using 30-40% of the CPU and ~ 9 GB of memory...like for a long time
I am still puzzled why I cannot see print() statements that are placed as the first line of .create_mesh(). I am confident is with the call to triangle but I want to be able to see it to confirm it.

Try writing to a file (open in append mode, write, close, and flush it).  Or setup python logger to write to file.  Maybe print doesn't show up due to buffering?

@Agent6-6-6 Ah! I think I have got it. It seems that the align_center() and align_to() problems are causing the meshing problem.
In the first instance of geometries not touching and shear center result being incorrect, there is a 0.0000000000001 (12 zeros) difference in geometry points where that value represents a gap.
In the second instance with the meshing problems, there is a ~ 0.000000000001 (11 zeros) overlap between the geometries which means that triangle is trying to create a super-small mesh to fill in that overlap sliver.
The reason there is not a problem on my example is because I am not aligning the geometries to the centroid. I have found that when aligning a geometry with the shapely.geometry.Polygon.centroid method and the shapely.affinity.translate() functions (i.e. get the centroid and translate the geometry to the origin) the resulting placement will never place the shape with it's centroid at (0, 0), it will only get to within 14-15 decimal places. A discrepancy small enough to not cause any meaningful errors on the calculation of the plastic centroid but small enough to create a meshing problem.

@Spectre5 Will try a logging approach. I am still stumped by this. Will let you know how it goes when I try later this eve.

Maybe there should be a tolerance input and then all points get rounded to that tolerance just before sending to triangle?  Surely none of us are working to the 12th digit of precision :)

...Aaand fixed!
@Spectre5, I implemented your suggestion. The Geometry class now carries a .tol attribute that is set to 12 by default. The value of the attribute represents the number of decimal places of precision to round point coordinates to. The rounding occurs upon instantiation of the Geometry instance during the .compile_geometry() method call. Specifically in the function create_points_and_facets().
@Agent6-6-6 I ran both of your examples that caused problems (shear center and hanging mesh). This change fixed both of them. Your script runs without issue now. Note: that in the shear center example, the SVG still comes out green (which is a bit of a puzzler for me since I confirmed that the points are landing on the exact same y-ordinate) but the meshing and shear centers come out correct. So it seems that shapely recognizes it as a "valid" geometry still and so does section-properties.
It is in the latest pushed commit.

@connorferster, good stuff, easy solution vs trying to debug further, especially if you're coming up against floating point precision issues (which I don't think you'll really solve any other way).
I seem to remember Robbies original clean_geometry module had a similar precision tolerance to merge points and rearrange the mesh if required (discussed and fixed in a prior issue I believe).
I seem to remember now coming across a particular issue previously when using shapely ages ago, when unpacking/unzipping a geometry to x/y coordinates lists in python, some of the values changed slightly from the value in the shapely geometry vs the X and Y unzipped coordinates (probably due to precision issues), I never really looked into it further as I did the same thing rounding to get things to agree so I could compare values after doing some further manipulations.
I guess this highlights the prior discussion we had on having some inbuilt warning regarding when your geometry isn't actually valid, because there is no warning currently unless you are particularly onto it with reviewing the properties and/or plots and realising something went wrong. Even just throwing an exception would stop potentially incorrect results from being used in production. I'm a big fan of protecting the user from themselves in this respect. Hopefully though this change puts all the little niggly misalignments to rest, and any gross misalignments or lack of joining can be observed with plot_geometry!
In the latest push there is still all the additional print statements that I think you added for temporary debugging.
A question, can you actually access the red/green plots you're showing from a script *.py file to check for the validity of the geometry?

@connorferster, I wonder to let shapely deal with it, could the shapely.ops.snap() be used to the same effect with a suitable tolerance. Just curious really if it achieves the same by tidying up geometry at small tolerances?
https://shapely.readthedocs.io/en/stable/manual.html#snapping

@Agent6-6-6 I am just reviewing the shapely snapping. The function "snaps the vertices in one geometry to the vertices in another". In this example snapping would not help us since we are essentially snapping to a facet. It seems that rounding to a tolerance works for both of these scenarios though.
Note: After implementing the tolerance, I ran the tests and noticed three of the calculations in the test suite were off in the seventh sig fig. I adjusted the test values so tests would pass but made note in the file which values were adjusted and their original value. The original values were from v1.0.8. The affected tests are in test_sections.py for the angle section. Please review and let me know your thoughts on this.
I implemented the tolerance on all Geometry but perhaps I should only implement it for CompoundGeometry?
I have come around now to your way of thinking about notifying the user if there is an issue with their geometry even after we implement techniques for reducing invalids. For disjoint geometry, I think a warning would be appropriate since the disjoint-ism only affects the warping properties calculations.

And yes, I agree it would be cool to show the status with line colours/weights on the plot_geometry method.

@connorferster, I came across one situation today where the creation of a compound geometry wasn't recognising the creation of a small internal hole that was created by moving/aligning together 4 separate geometries creating a small internal rectangle.
Out of town at the moment for a few days, but will post example on return when back at computer. I managed to get it to work eventually by shifting section vs aligning and changing around the order of some of the shift/mirror/align manipulations. So possibly some more precision issues.
Will try boil down to a simple example to demonstrate if I can when I return.

Hi @connorferster
Here's a contrived example showing how two methods to achieve the same thing regarding creating and moving the tee section results in the hole being created either being recognised or not recognised. Appreciate if you could have a look.
Change the which_option variable between 1 or 2 to see the two behaviours.
import sectionproperties.pre.sections as sections
from sectionproperties.analysis.cross_section import Section

which_option = 2  # 1 or 2

d = 612  # I section depth
b_f = 229  # I-section flange width
t_f = 19.6  # I-section flange thickness
t_w = 11.9  # I-section web thickness
r_1 = 14  # I-section root radii

d_tee = 202  # strengthening tee depth
b_tee = 178  # strengthening tee flange width
t_f_tee = 12.8  # strenghtening tee flange thickness
t_w_tee = 7.8  # strengthening tee web thickness
r_1_tee = 11.4  # strnegthening tee root radii

gap = 2  # gap between section and tee

weld_size = 6  # weld size

n_r = 50  # number of points considered around root radii

mesh_area = 10  # max mesh size

# weld base geometry
weld = sections.Geometry.from_points(points=[[0, 0], [weld_size, 0], [0, weld_size]], facets=[[0, 1], [1, 2], [2, 0]])

# I-Section beam
geometry1 = sections.i_section(d=d, b=b_f, t_f=t_f, t_w=t_w, r=r_1, n_r=n_r).align_center()  # shift_section(-b_f / 2, -d / 2)

# strengthening tee
if which_option == 1:
    # ---- OPTION 1 - HOLE CREATED GETS RECOGNISED
    geometry2 = (
        sections.tee_section(d=d_tee, b=b_tee, t_f=t_f_tee, t_w=t_w_tee, r=r_1_tee, n_r=n_r)
        .mirror_section(axis='x', mirror_point=[0, 0])
        .shift_section(-b_tee / 2, -d / 2 - gap)
    )
elif which_option == 2:
    # ---- OPTION 2 - HOLE CREATED DOES NOT GET RECOGNISED
    geometry2 = (
        sections.tee_section(d=d_tee, b=b_tee, t_f=t_f_tee, t_w=t_w_tee, r=r_1_tee, n_r=n_r)
        .mirror_section(axis='x', mirror_point=[0, 0])
        .align_center()
        .align_to(geometry1, 'top')
        .shift_section(0, -gap)
    )

# weld 1 & 2
geometry3 = weld.mirror_section(axis='x', mirror_point=[0, 0])
geometry3 = geometry3.shift_section(t_w_tee / 2, -d / 2)
geometry4 = geometry3.mirror_section(axis='y', mirror_point=[0, 0])

# total number of individual geometry elements
geo_number = 4

# assemble geometry list
geo_list = [globals()[f'geometry{i}'] for i in range(1, geo_number + 1)]

# create merged section
geometry = sections.CompoundGeometry(geo_list)

geometry.plot_geometry()

# CREATE MESH
geometry.create_mesh(mesh_sizes=[mesh_area])

# create section
section = Section(geometry=geometry, time_info=True)

# calculate results
section.calculate_geometric_properties()
section.calculate_plastic_properties()
section.calculate_warping_properties()

# plot results
section.plot_mesh()
section.plot_centroids()

# display all results
# check https://sectionproperties.readthedocs.io/en/latest/rst/post.html for definitions
section.display_results(fmt='.3f')

Note there is no longer an add_hole procedure, so even if you recognise the issue you cannot manually correct it. But unsure why it is actually occurring, especially after you added in the tolerance rounding (unsure if that's at play here).

Hi @Agent6-6-6, I am so glad you are finding these bugs. I hope to be able to fix this one but, if not, it can at least be addressed in the documentation.
The problem in this instance seems to be the geometry1.align_center(). Aligning compound geometries to the origin seems to create floating point errors in ways that don't seem to crop up otherwise. For example, if you replace the .align_center() on geometry1 and restore the .shift_section(...) that you have commented beside it, then the problem goes away, too.
The reason the .tol feature does not fix this issue is because the .tol only applies to geometry information that is passed into the Section. By introducing .tol in the way that I have, there is now a disconnect between the shapely geometry and the geometry data (i.e. .points, .facets, etc) that is used for calculating the section properties.
It would be good if that disconnect did not exist and for the .tol to apply to both the shapely geometry at instantiation and the analysis geometry. I am going to play around with this and see what I can come up with.

It's also worth mentioning that, for this problem, an .add_hole() method would not fix it simply because the gap is not a fully enclosed space. The way that holes are recognized in CompoundGeometry is by performing a union with all of the constituent shapes in the CompoundGeometry. Once the union is complete, I simply query how many interior regions are now found in the unified shape. Because I trust shapely, I think that if there has been an interior region created, it will always be recognized. In this case though, no interior region has been created due to these floating point errors.

@connorferster, I see putting in a few print statements in align_center() that you indeed get some minor little bits added on.
I can solve the issue by rounding the centroid variables cx, cy to a tolerance in align_center() method. Unsure if that affects anything else though with other examples when centroid isn't at zero/zero point. The method is also called by the calculation of plastic properties for example and have not looked at what it's doing in there.

@Agent6-6-6 Great idea! I have implemented it and it fixes the problem in the script you provided and it passes all of the tests.

@connorferster, a few more small niggles I found while just messing around trying to see if I could break things :-
item 1
This example results in the code displaying the plot geometry, then it exits with no exception. Suspect it's another precision issue with rotating about [0, 0] point, if you change to [0.00000001, 0] for example it will complete without exiting with no warning?
import sectionproperties.pre.sections as sections
from sectionproperties.analysis.cross_section import Section

geometry_1 = sections.i_section(d=203, b=133, t_f=7.8, t_w=5.8, r=8.9, n_r=8)
geometry_2 = geometry_1.align_center(geometry_1).align_to(geometry_1, 'top').rotate_section(angle=90, rot_point=[0, 0])
geometry_3 = geometry_1.align_center(geometry_1).align_to(geometry_1, 'top').rotate_section(angle=45, rot_point=[0, 0])
compound = geometry_1 + geometry_2 + geometry_3
compound.plot_geometry()
mesh_area = 3
compound.create_mesh(mesh_sizes=[mesh_area])
section = Section(geometry=compound, time_info=True)
section.plot_mesh()

item 2
This example results in an unhandled exception in plot_geometry (note have just changed the alignment to 'bottom' for geometry_3, otherwise the same code as item 1 above. Not quite sure what is occurring here but may also be precision related?
import sectionproperties.pre.sections as sections
from sectionproperties.analysis.cross_section import Section

geometry_1 = sections.i_section(d=203, b=133, t_f=7.8, t_w=5.8, r=8.9, n_r=8)
geometry_2 = geometry_1.align_center(geometry_1).align_to(geometry_1, 'top').rotate_section(angle=90, rot_point=[0, 0])
geometry_3 = geometry_1.align_center(geometry_1).align_to(geometry_1, 'bottom').rotate_section(angle=45, rot_point=[0, 0])
compound = geometry_1 + geometry_2 + geometry_3
compound.plot_geometry()
mesh_area = 3
compound.create_mesh(mesh_sizes=[mesh_area])
section = Section(geometry=compound, time_info=True)
section.plot_mesh()

Error:-
Traceback (most recent call last):
  File "....................shapely-section-properties/example.py", line 13, in <module>
    compound.plot_geometry()  # note custom control points
  File "....................shapely-section-properties\sectionproperties\pre\sections.py", line 621, in plot_geometry
    ax.plot(h[0], h[1], "rx", markersize=5, label="Holes")
IndexError: tuple index out of range

Interesting...looking forward to checking these out!

@Agent6-6-6 Took a look a these. For Item 1, I was not able to reproduce. I was able to copy-paste the example you gave and it would run without interruption.
Item 2, though, was a coding error on my part. I thought I was getting a tuple from shapely but it was giving me a nested tuple. Easy fix.
While I was in there, though, I realized that I had some errors in Geometry.align_to() and the doc-string was incomplete. Also decided it would be very useful to be able to change the size/resolution of the resulting figure from Geometry.plot_geometry() so I implemented that, also.
Next, implement these persistent control_points and continue updating the Sphinx docs.

@connorferster, for item 1, this now works for me with your latest commits. Guessing your fixes to align_to module fixed something  along the way that was causing the issue 👍.


package controller_manager not found error

ros2 launch lidarbot_bringup lidarbot_bringup_launch.py

After running above command I got error controller_manager package not found

Yes the above error was solve but the problem is that while running command
rviz2 -d src/lidarbot_slam/rviz/lidarbot_slam.rviz
I got the new error on my development machine i.e.
qt.qpa.xcb: could not connect to display localhost:11.0
qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: eglfs, linuxfb, minimal, minimalegl, offscreen, vnc, xcb.```

I haven't come across this error before, but it seems like it might a problem with the QT software install. To confirm this, try opening rviz on its own in the terminal with:
rviz2


Fix Clusterrole issue

Issue #, if available:
The issue is when we execute 03_02_Distributed_Training_MPI_Horovod notebook, the !kubectl apply -f distributed-training-jobs/distributed-mpi-job.yaml returns error:
User "system:serviceaccount:kubeflow:default-editor" cannot create resource "mpijobs" in the API group "kubeflow.org" in the namespace "kubeflow"

Description of changes:
Edit sa default-editor to have the privilege to create resource in "kubeflow.org"
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@PatrickXYS remember to have clear commit message next time